bpwwer

The setup looks correct. If PG3 is displaying a value and that value is changing from day to day, then the node server is working correctly. The node server never talks to the ISY directly, everything goes through PG3. So if PG3 has the right value and the ISY doesn't, it's not an issue with the node server. You didn't mention any other problems, but if it's just that one value that isn't getting sent to the ISY, that would be very, very strange. The only thing I can think of that might cause something like that would be if the node server's profile file on the ISY was missing the ETo definition, but the AC does get the proper definition. You can try re-sending the profile files to the ISY (PG3, node server details, upload profile button) and see if that helps. Otherwise, I'd suggest checking the node server log and look for when it sends the ETo value to PG3, should happen within a minute or so of midnight. Using the timestamp from that, check the PG3 log for the same time and see what it does with it. There should be log entries there showing it sending the value on to the ISY. Then at least we know if the value is making it to the ISY or not.

ETo is set to zero if there isn't any forecast data. The node server will enable the ETo calculation only if the custom parameter "Forecast" value matches a station ID returned when it queries the WeatherFlow servers for station meta data.

Unless you have a backup of Polyglot, you can't. The information about node servers is stored in a database, when you factory reset the Polisy, you removed that database. If you have a Polyglot backup, restore it and you should be good. Otherwise, you'll have manually remove the node servers from the ISY (node servers -> configuration -> slot #, and click delete) and then re-install them from Polyglot.

The query isn't returning data in the format expected. If you turn on debug level logging, the log will show the actual URL that is being used to query AERIS. Note, that URL will contain your secret API key. But I'd need to see the output from that query to know specifically why it's failing.

There was a bug about 6 months back that set it to zero after the first day, but that should be fixed. I see mine is at zero also so I'll do some debugging. It does need forecast information to calculate ETo and it does the calculation at midnight using the previous day's information.

Check the log, it will log any errors encountered while querying for that data.

I only suggest that to try and determine if there is something specific your doing or using that causes this. It may well be that the ISY is just overloaded. It has limited processing power and fairly limited TCP/IP stack. The IoP will certainly help as it has significantly more processing power. I have a lot of influence over what is done in PG3 as I'm the only developer working on it. But not much influence over the ISY software. #1 PG3 does support the ability for a node server to restart itself as that's currently the only way it can be implemented. The ISY doesn't have any built in methods to control node servers and it would be a pretty big project to add that. We can discuss this, but I'll make no promises. #2 is a possibility. Right now, I don't know of anyone else having problems with the current startup process and there are folks that run quite a few node servers. I've had near 25 starting and have not seen the issues you have. Of course this is with IoP with no programs running on it. Because node servers can vary quite a bit in how long they take to initialize, it may be hard to tune something like this. It can't be too long, a couple of minutes and you could have systems that take 30-60 minutes to start. A few seconds may or may not help. #3 is already being done. PG3 has both a per-node server queue and a global queue for messages being sent to the ISY with retries if they fail. In your case, the ISY either doesn't respond (which will cause PG3 to give up) or it is responding with an error that indicates a retry wouldn't help. You log did not show any cases where PG3 was retrying requests. It didn't look like there was enough traffic to trigger throttling. I do understand about all the programming needed to deal with HVAC. For a while I had a lot of programming for mine that took into account windows being open, inside vs. outside temperature diff, etc. The data was coming from WeatherFlow, my alarm sensors, and the thermostat. It controlled the thermostat and an whole house fan. I disabling most of it at one point when a window sensor failed and I couldn't get the whole house fan to turn off.

So I'm clear, the only thing running on the Polisy right now is PG3? I'm assuming you did some experiments that prompted you to set up the Polisy reboot when the ISY stops/starts, why? What happens if you don't? PG3 shouldn't care if the ISY stops or crashes, other than it will fail to get info from it and fail to send information to it. I don't believe it opens any connections to the ISY that it expects to say open until it is restarted. If it's not able to restore operation when the ISY comes back up, that would be a bug in PG3. Clearly data is being sent to the ISY, the event viewer log shows that as do the node server and PG3 logs. If the ISY isn't doing anything with that data, then it's a problem with the ISY. As I mentioned previous, the PG3 log shows a lot of failures communicating with the ISY when it starts up. When PG3 starts, it starts all the node servers which can generate a quite a bit of traffic as it tries to make sure the ISY is fully up to date with each node server's current status. With all the random failures it's getting, it's not really surprising that you're having issues. That would explain why restarting the node server when things are a lot less busy allows it to fully update the ISY and work. You obviously have a complicated setup on the ISY which means it's going to be difficult or impossible for anyone to replicate what you're seeing. There are a few things you could try that may help narrow down the problem. 1. You could stop all but one node server and restart PG3 and see if you still get the communication failures on startup. (If you manually stop a node server, it won't automatically restart when PG3 starts). You could try this with each node server, maybe only one really impacts the ISY. 2. With a good backup of the ISY, you could try disabling or removing all programs and then restart PG3 and see if it still has failures starting. Maybe a program or set of programs is causing so much load that it can't handle starting multiple node servers. Based on what I've seen in the logs it seems like the ISY is just overloaded and there may not be anything anyone can do about that.

I just saw you posted the PG3 log in another thread so I can provide more updates here. Looking at the PG3 log. At 10:50:52 PG3 scheduled the request to send the temperature value to the ISY: 7/21/2022, 10:50:52 [pg3] debug: Scheduling http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930073475/report/status/CLITEMP/23.6/4 to group statusGroup It got a response back from the ISY indicating it was successful 7/21/2022, 10:50:52 [pg3] debug: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [200] :: 1526.422921ms - http://192.168.200.251:80/rest/ns/4/nodes/n004_controller/report/status/CLITEMP/26.93/4 So again, I see no indication that anything is wrong. PG3 is getting updates from the OWM node server and passing those on to the ISY successfuly. I'm confused as the PG3 shows PG3 communicating just fine with the ISY until 02:13:52, that's the first time a request from PG3 to the ISY failed. It continued failing until 09:53:05 when it looks like the Polisy was rebooted. PG3 restarted started at 10:10:33 At 10:10:54 PG3 sends some info to the ISY and the ISY responds with success. At 10:10:57 PG3 sends a /rest/nodes request and the ISY fails to respond. But then other requests after that are successful. At 10:11:07 PG3 tries to add the OWM controller node to the ISY, this fails with a "bad request" error, however, it's not a bad request. This continues for about a minute with various "bad request" responses, successful responses and no responses to the PG3. This is all happening while node servers are trying to initializing. It appears that the ISY is struggling to keep up. It mostly clears up after that, but the ISY is still not responding to the /rest/nodes requests from PG3. For the next few minutes there brief periods where the ISY fails to respond to PG3 but then it starts responding normally. Probably as node servers finish their initialization. From that point everything is normal until it is restarted at 11:03. After the restart, I see the same errors happening where the ISY fails to respond to PG3. PG3 restarts place a heavy load on the ISY as node servers start and initialize. On your system, it seems like it takes 5-10 minutes after a PG3 restart before the load on the ISY decreases to what it can handle.

Did you reload the browser window after restarting the Polisy? The uptime value has been a pain because the browser seems to cache info related to that causing it to be wrong.

Thanks for providing such detailed information! Would it be possible to get the PG3 log file for the that time as well? You can PM it to me if you don't want to post that. From the node server log. Until 09:50:05, the node server was working properly and sending data to PG3. It was unaware that there were issues with the ISY. At this point, the node server stopped (I'm guessing because you power cycled the Polisy. The node server starts again at 10:10:54. There are no issues with it starting and it runs and is sending data to PG3 until 10:50:59 when it is again stopped. I'm not sure how you are determining that no data was sent. At 10:50:52 - 10:50:59 the log shows it sending: Successfully set controller :: CLITEMP to 26.93 UOM 4 Successfully set controller :: GV2 to 28.58 UOM 4 Successfully set controller :: DEWPT to 20.52 UOM 4 Successfully set forecast_0 :: DEWPT to 20.3 UOM 4 Successfully set forecast_0 :: GV0 to 27.0 UOM 4 Successfully set forecast_0 :: GV20 to 3.12 UOM 106 Successfully set forecast_0 :: GV2 to 26.0 UOM 4 This says the node server successfully sent those updates to PG3. With the info I have, I can't tell you what PG3 did with them. Also, the node server only sends values that changed. In this case, for the main node it was only the temperature, dewpoint and feels like temperature that changed so that's what was sent. How often stuff changes is really dependent on how often OpenWeatherMap updates the info for your location, not your node server polling intervals. The node server starts again at 11:03:29. And again, there are no issues with it starting and running. From the log, the node server was running properly the whole time. I see no indication that a restart was needed.

If it wasn't obvious, my test and maths seemed to disprove my original theory that the AC was the cause of slow program loading. For the 88 programs, the time was split almost evenly between waiting for the ISY to prepare the data and the actual download of the data. I used the developers tools in the browser to get the timing info for the /rest/programs request from the ISY. So I just tried the same thing with IOP running on a PolisyPro. Unfortunately, I don't have as many programs on that so I'm not sure this is a valid comparison. But it was many times faster. 26 programs in 4.5ms, again split fairly evenly between processing time and content download time. If the time scales linearly (a big assumption), it would be able to send 980 programs in about 170ms. That's like 20x what I calculated for the i994. These tests were done between the device and my browser over a wired ethernet connection. Since PG3 can work with both IOP and i994 ISY's, it communicates with them over the network interface. It may be a bit more optimized if it's using the localhost address, but it still uses the network drivers. However, that may be offset by the fact that it's one network driver that's handling both ends of the communication (twice as much work) when both IOP and PG3 are running on a single Polisy.

I'm glad that UDI has reviewed the issue and at least has some idea of the cause. 980 programs is a lot of programs to format and send out and I new the AC was slow about loading programs, but I thought that was because the AC had to process them, more than the ISY having to send them. It can be difficult to understand the effects of scale for something like this without doing the math. 3 minutes seemed like a long time so I did a quick test. I only have 88 programs but it takes about 300ms to send that. So scaling that up to 980 and I get close to 3.5 minutes. It will probably take more than just compressing the data to get reasonable performance with 2000 programs. I'm thinking it will need to reduce the size of the data by 20x or more. But looking at the data, it may be possible simply by reformatting the data into something far less verbose. An interesting problem for sure. Thanks for the effort to help resolve it.

@johnnytThanks for the well written response. You make some very good points! In my defense, I didn't have anything to do with the original design of PG2 or PG3. I originally signed up to help test PG3 and now I'm the only developer working on it. There's enough work that I do have a tendency to try and avoid major design changes if I can. My concerns are specifically around adding the capability to automatically reboot/restart. Once you can automate a work-a-round for a problem, it tends to not be a problem for you anymore. Sure, no one would ask for a weekly restart, but if it happens and you don't really know it happened are you really going to care? I specifically didn't say anything about power outages because the system should recover from those without any user intervention. If it isn't, we really need to understand why. I'm not completely opposed to automatic restarts. PG3 will automatically restart if it crashes and a lot of work has gone into make it recover and continue if it does crash. There's not really anything you can set up that would improve the process. Eventually, we should get to the same point with Node server, but we're a long way away from that now. Keep in mind that the production release of PG3 is just a little over 2 months old. It can take many iterations of a software product to work out the issues. With the number of different node servers and use cases, it is impossible for us to test even a small fraction of what it's capable of doing. I've been watching your other thread on ISY-Inventory with interest because I do want to know if there's anything we need to do in PG3 to prevent it from happening again. The key is to figure out what is really happening. Node server should not be able to crash an ISY. I do sometimes do stress testing with PG3 to see if I can make bad things happen. It's been a while since I've done any of that with a i994. It is possible to overload the i994's network interface, but I've not seen that crash the ISY. If you haven't, I highly recommend you submit a trouble ticket to UDI for the ISY crashes. Neither @simplextechor I have access to ISY code to debug it. Best we can do is help to define a reproducible test case.

Because when you restart the ISY the ISY forgets everything it's been told about nodes. The query on restart is a work-around to try and restore everything's current state. The connection status is kind of a hack to provide some indication of the node server state. The ISY was designed to monitor the state of nodes, not the the state of the node's controller. I.E. where is the Insteon on-line status or the z-wave on-line status? So basically, the ISY doesn't know the on-line connection status until PG3 sends it an update, which it won't do until the status changes. PG3 will continue to get updates. One of the things we're looking at is adding more status communication between node servers and PG3 and better error reporting for when things do go wrong. But since the ISY doesn't have any support for this, I'm not sure how this will be exposed to users yet. Also, none of these components were designed to be restarted. I understand the desire to try and automatically recover from catastrophic events, but just how often are you experiencing these? I know we'll never get to 100% uptime, but events that require a restart/reboot should be very rare. And, yeah, I know we're not there yet. But if everyone simply reboots or restarts automatically whenever there's an issue and the issue doesn't get reported, we'll never get there because developers won't know there's an issue to fix. You can restart a node server by clicking the restart button in PG3 you can restart PG3 via a menu option. Maybe I'm just lucky. But my production polyglot instance has been up for over 3 months which is how long the Polisy box has been running since I last cut power to it while doing electrical work. Yes, I do occasionally have issues with node servers and most of time it is because the device the node server is for has issues and the node server doesn't handle the device failure very well.

The on-line status isn't doing what you probably think it does. That value represents the status of the connection between the node server and PG3 only. It is set only when the status of that connection changes. When the node server is stopped, it is set to 'disconnected'. If the node server crashes and drops the connection, it is set to 'failed'. When the node server starts it is set to 'connected'. The connection status does not indicate the node server's status beyond the connection to PG3, it is also not any kind of heartbeat indicator. This node server's function is to query the OWM server for weather data and pass that data on to the ISY. A query action would logically force it to query the OWM sever immediately instead of waiting for the next query interval. But that's not what you are really asking for. I believe you're asking for the ability to query the status of the node server itself. But that doesn't really work. The node server could only ever report that it is running. If it's not running, it can't report that it's not running. I suspect that what you want is the ability to query PG3 for the status of the node server, but the ISY doesn't have a way to do that.

I can't speak for the original developer of this node server, but the answer to does UDI ever pick up unmaintained node servers is sometimes. Other individual developers can also pick up and maintain a node server. The pool of node server developers is not real large and in many cases, the developer has created a node server because they need the node server. Without having the equipment, it can be difficult for another developer to take over the maintenance of a node server.

That's not entirely true. More than 50% of the PG3 node servers are installed from the source in a github repository and even some that are not installed from a source repository still make the source available. This may not always be true because unfortunately, we live in a world where people will take advantage of other's work and deprive node server authors of reasonable compensation for their work. Also, there are companies that aren't willing to support node servers for their products because they're afraid the lack of protection on the node server source could expose something they consider secret.

Don't be embarrassed, like I said, PG3 doesn't tell you that it failed to install it to the ISY/IOP and makes you think everything is fine. This is one of many areas where it doesn't do a good job of communicating issues.

Nothing with your current version of PG3. The ISY record in the PG3 database must have a uuid, that's the key it uses to edit or remove the entry. Without the uuid, it is unable to do anything with that record. The latest version of PG3 should automatically remove that invalid entry for you. The current PG3 release is 3.0.63, if you don't have that, you need to upgrade.

PG3 (the current version anyway) will happily install node servers even if it can't communicate with the ISY/IoP. So when node servers fail to show up in the AC, it almost always means you haven't entered the correct username/password for PG3 to authenticate with the ISY/IoP. Go to the ISYS menu in PG3 and select the Edit Current ISY option. Verify the information there. It should have the either the IOP's IP address or 127.0.0.1 or localhost, I recommend either actual IP address or 127.0.0.1. Verify the port, the default for IOP is 8080. Also verify http is selected and not https. https is only valid for very special cases. And, of course, verify the username and password are the same as what you use to log into the AC.

It doesn't. The WeatherFlow hub uses UDP broadcasts to push it'd data to the local network. The node server just listens for those broadcasts. I don't know anything about VLAN's and their configuration so I don't know how or if UDP broadcast packets would route through them. The WeatherFlow hub doesn't have a way to connect to it directly via a TCP/IP address.

That's not a simple request for a number of reasons. The node server's primary operating mode is to use the data directly from the sensors. NearCast values are available from the server only so it can't be used with the node server's primary mode. The node server does use the server data when first started to get historical accumulation, but in that case it also wouldn't really make sense since it would then be mixing NearCast values with non-NearCast values in the values reported to the ISY. It would be possible to use the NearCast values when the node server is in the remote data mode, but that would mean differences between local and remote, would probably need a configuration to enable or disable and would make the code a lot more complex. Given that NearCast isn't available everywhere, I don't think it makes sense to add. At least not at this time.

From what I see in the log, the node server is working correctly. It's not reporting any data to the ISY because it's not receiving any data from your hub. Either the hub isn't working or something on your local network is preventing the hub's data broadcasts from reaching the node server.