lots of errors at startup

[johnnyt]

Hi,

I'm using 994i and see lots of PG3 errors and encounter general struggles whenever I restart a NS, and especially so when I restart Polisy which restarts *all* the NS at once.

For my testing and log capture today, I first made sure things were stable and ISY was not particularly busy. I then restarted just WeatherFlow NS. I did same with Airthings NS (with a break in between).

Attached are PG3 logs showing entries immediately following the WeatherFlow restart, along with the NS logs for it for the same period. Also attached are the PG3 logs for Airthings to show how both have some similar struggles (> 5000ms, retries, aborts, etc.) I removed (substituted) my ISY ID and any references to API keys thinking it's not a good idea to broadcast that info.

For the record, I do always have a number programs running on ISY but they are just 60 sec. counters, which, hopefully is not consuming much ISY CPU but if that's the case (and I'm really wondering if it is) let me know because I always have some counters running, e.g.

If
        'HVAC / Venting-Humidity / 1-Fan High Relay' Status is On
Then
        Repeat Every  1 minute 
           $iHVAC.Count.FanHigh += 1
 

I also made sure to put some "wait" commands in any ISY programs that act on Weatherflow data to at least spread the load out on the ISY side. I'd rather be able to throttle the NS/PG3 if that was possible.

What else can I do? Do all the errors/delays suggest a busy ISY or is there something else, like network collisions or something, that could be causing them?

I think I asked a while back and the answer was 'no' but in latest version of PG3 is there a way to control NS startup, e.g. set order and delay between them, and/or a way to stop/start NS from ISY? Is that something that could be added in the future?

Any info would be appreciated

WeatherFlowLogs.zip

[bpwwer]

Based on the errors I'd say you have some significant network issues.

This really has nothing to do with PG3 or the node servers.  PG3 does have built in limits to how much network traffic it can generate to the ISY and with what you're doing by simply starting a single node server in no way comes close to any of those limits.  

Just for reference, my test environment currently has 30 node servers installed and I'm constantly stopping and restarting PG3 and I don't see any issues like this.

When PG3 is trying to communicate with the ISY it will try up to 3 times for each request before giving up.  In you airthings log we see:

11/14/2022, 11:03:58 [pg3] warn: ISY Response: [Try: 1] [<my 994i ID>] :: [HPE_INVALID_CONSTANT] :: 4949.559136ms -http://192.168.200.251:80/rest/nodes
11/14/2022, 11:04:00 [pg3] warn: ISY Response: [Try: 2] [<my 994i ID>] :: [HPE_INVALID_CONSTANT] :: 1991.814428ms -http://192.168.200.251:80/rest/nodes
11/14/2022, 11:04:05 [pg3] warn: ISY Response: [Try: 3] [<my 994i ID>] :: [ECONNABORTED] :: 5012.364673ms -http://192.168.200.251:80/rest/nodes
11/14/2022, 11:04:10 [pg3] error: ISY Response: [MAX TRIES EXCEEDED] [<my 994i ID>] :: [ECONNABORTED] :: 5014.965651ms -http://192.168.200.251:80/rest/nodes

The HPE_INVALID_CONSTANT error means that PG3 got back a corrupted response from the ISY.  It also took about 5 seconds to get that.

I'm not sure what ECONNABORTED means specifically as it's not something I normally see.  But it does mean the connection between PG3 and ISY was aborted for some reason.

We see that each try is taking quite a while. 5 seconds, 2 seconds, 5 seconds.  Normal times would be 5-100ms 

I will occasionally see ECONNRESET messages which mean the ISY is too busy to handle the request and then on try 2, the request will succeed. 

All of this points to either a really bad network connection, a failing network device (router/switch) or a failing ISY.

[johnnyt]

I'll play with network path to see if there's an issue with switches connecting 994i and Polisy to router (both are on same switch now). I should mentioned that I often get sub-100ms responses so it's not clear to me I'm having a general network issue.

In the meantime if you don't know what ECONNABORTED who would? Cause I get those all the time.

Also I just upgraded PG3 from 3.1.12 to 3.1.14, which required a PG3 restart. Below is a chunk of the PG3 logs I got at startup. There are a number of unhandled errors, a couple "REPORT THIS" messages, several driver errors, and, of course the pretty normal "ECCONNABORTED" messages.

Other than the unhandled errors, would the other errors be due to network problems or failing ISY, or could there be something else going on? If changing switches doesn't fix the problem I will reach out to UDI support but want to be sure I've ruled other possibilities out.

Node 6 is WeatherFlow NS and node 8 is Airthings NS. Attached is screenshot of the PG3 Dashboard for my 994i

 

11/15/2022, 09:43:57 [pg3] error: Error: GET http://192.168.200.251:80/rest/nodes Failed :: ECONNABORTED
11/15/2022, 09:43:57 [pg3] error: [<my ISY ID>_6] checkDrivers failed :: TypeError: Cannot read properties of undefined (reading 'data')
    at Object.checkDrivers (/var/polyglot/node_modules/@universaldevices/pg3/lib/modules/Node server/status.js:269:64)
    at async /var/polyglot/node_modules/@universaldevices/pg3/lib/modules/Node server/command.js:144:13
    at async Promise.all (index 0)
11/15/2022, 09:43:57 [pg3] info: [<my ISY ID>_6] adding node forecast_8 to ISY
11/15/2022, 09:44:02 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 5020.398338ms - http://192.168.200.251:80/rest/ns/8/nodes/n008_controller/report/status/GV2/1/56
11/15/2022, 09:44:07 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 10003.68036ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_8/add/forecast?primary=n006_controller&name=Forecast%208
11/15/2022, 09:44:10 [pg3] error: No code in error response
11/15/2022, 09:44:10 [pg3] warn: ISY Response: [Try: 2] [<my ISY ID>] :: [400 - OK] :: 2459.073208ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_8/add/forecast?primary=n006_controller&name=Forecast%208
11/15/2022, 09:44:10 [pg3] error: No code in error response
11/15/2022, 09:44:10 [pg3] warn: ISY Response: [Try: 3] [<my ISY ID>] :: [400 - OK] :: 42.763673ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_8/add/forecast?primary=n006_controller&name=Forecast%208
11/15/2022, 09:44:10 [pg3] error: No code in error response
11/15/2022, 09:44:10 [pg3] error: ISY Response: [MAX TRIES EXCEEDED] [<my ISY ID>] :: [400 - OK] :: 434.252744ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_8/add/forecast?primary=n006_controller&name=Forecast%208
11/15/2022, 09:44:10 [pg3] error: Error: GET http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_8/add/forecast?primary=n006_controller&name=Forecast%208 Failed :: 400 - OK
11/15/2022, 09:44:10 [pg3] info: [<my ISY ID>_6] addnode sucessfully added node forecast_8
11/15/2022, 09:44:10 [pg3] error: unhandledRejection REPORT THIS!: [object Promise], reason: TypeError: Cannot read properties of undefined (reading 'status')
11/15/2022, 09:44:10 [pg3] warn: node forecast_0 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_1 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_2 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_3 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_4 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_6 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_5 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_7 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node forecast_9 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node s_2930037297 on profile 8 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:11 [pg3] warn: node s_2930027997 on profile 8 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:12 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:44:12 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:44:12 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:44:12 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:44:12 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:44:12 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 7534.005203ms - http://192.168.200.251:80/rest/nodes/n006_207296
11/15/2022, 09:44:12 [pg3] warn: node 207296 on profile 6 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:17 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 5007.305041ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_forecast_9/report/status/GV1/-5.0/4
11/15/2022, 09:44:23 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 5020.107423ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:44:28 [pg3] warn: ISY Response: [Try: 2] [<my ISY ID>] :: [ECONNABORTED] :: 5027.978098ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:44:33 [pg3] warn: ISY Response: [Try: 3] [<my ISY ID>] :: [ECONNABORTED] :: 5015.460456ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:44:36 [pg3] warn: node s_2930071732 on profile 8 already exists, no nodeDef or driver changes detected
11/15/2022, 09:44:38 [pg3] error: ISY Response: [MAX TRIES EXCEEDED] [<my ISY ID>] :: [ECONNABORTED] :: 5010.138324ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:44:38 [pg3] error: Error: GET http://192.168.200.251:80/rest/nodes Failed :: ECONNABORTED
11/15/2022, 09:44:39 [pg3] error: [<my ISY ID>_8] checkDrivers failed :: TypeError: Cannot read properties of undefined (reading 'data')
    at Object.checkDrivers (/var/polyglot/node_modules/@universaldevices/pg3/lib/modules/Node server/status.js:269:64)
    at async /var/polyglot/node_modules/@universaldevices/pg3/lib/modules/Node server/command.js:144:13
    at async Promise.all (index 0)
11/15/2022, 09:44:39 [pg3] info: [<my ISY ID>_8] adding node s_2930029938 to ISY
11/15/2022, 09:44:41 [pg3] error: No code in error response
11/15/2022, 09:44:41 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [400 - OK] :: 2917.629801ms - http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930029938/add/Sensor?primary=n008_controller&name=Main%20Floor
11/15/2022, 09:44:41 [pg3] error: No code in error response
11/15/2022, 09:44:41 [pg3] warn: ISY Response: [Try: 2] [<my ISY ID>] :: [400 - OK] :: 17.414811ms - http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930029938/add/Sensor?primary=n008_controller&name=Main%20Floor
11/15/2022, 09:44:41 [pg3] error: No code in error response
11/15/2022, 09:44:42 [pg3] warn: ISY Response: [Try: 3] [<my ISY ID>] :: [400 - OK] :: 13.28028ms - http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930029938/add/Sensor?primary=n008_controller&name=Main%20Floor
11/15/2022, 09:44:42 [pg3] error: No code in error response
11/15/2022, 09:44:42 [pg3] error: ISY Response: [MAX TRIES EXCEEDED] [<my ISY ID>] :: [400 - OK] :: 35.297172ms - http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930029938/add/Sensor?primary=n008_controller&name=Main%20Floor
11/15/2022, 09:44:42 [pg3] error: Error: GET http://192.168.200.251:80/rest/ns/8/nodes/n008_s_2930029938/add/Sensor?primary=n008_controller&name=Main%20Floor Failed :: 400 - OK
11/15/2022, 09:44:42 [pg3] info: [<my ISY ID>_8] addnode sucessfully added node s_2930029938
11/15/2022, 09:44:42 [pg3] error: unhandledRejection REPORT THIS!: [object Promise], reason: TypeError: Cannot read properties of undefined (reading 'status')
11/15/2022, 09:44:55 [pg3] warn: ISY Response: [Try: 1] [<my ISY ID>] :: [ECONNABORTED] :: 5004.806641ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:45:00 [pg3] warn: ISY Response: [Try: 2] [<my ISY ID>] :: [ECONNABORTED] :: 5007.780992ms - http://192.168.200.251:80/rest/nodes
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver
11/15/2022, 09:45:01 [pg3] error: addnode add driver error: UNIQUE constraint failed: driver.uuid, driver.profileNum, driver.address, driver.driver

 

 

 

[bpwwer]

All of the errors in your log are because of poor network connectivity with the ISY.  PG3 is trying really hard to communicate with the ISY but there are just too many failures so things aren't going to work very well.

I'm not a network engineer nor do I have experience with network device drivers.  The ECONNABORT is something failing inside the OS level network driver.  I don't know what specifically can trigger that. Possibly it is timing out waiting for packets or maybe getting corrupted packets.  Googling ECONNABORT didn't really provide any clarity.

I will say that don't ignore any component of the network path.  I once had issues that I traced to a network cable having gone bad and never did figure out why it went bad.  

When the ISY is too busy to respond you'll get ECONNRESET errors and those are temporary and the second try by PG3 will then succeed.  So all I can really tell you is that this is something else.

[johnnyt]

So I moved the 994i and Polisy off the unmanaged switch I was using and connected them directly to my main managed switch. I then ran a cable test and it returned "normal" (ports 11 and 12 below). Port 9 shown is connection to unmanaged switch so cables being used aren't the problem.

 

I also don't see any Tx / Rx errors on those ports on the switch monitoring page.

Finally, I did some packet capturing at router of 994i and Polisy routing calls and traffic to/from my PC to them and there were no errors reported.

I'm still, however, getting roughly the same errors in PG3 when I stop/start WeatherFlow NS after the switch changes.

So I think I'm now trying to figure out whether 994i or Polisy NIC might be struggling/failing. Is there a tool I can use at Polisy OS level to test the Polisy NIC to see if it's failing? I don't believe there's anything I can run on 994i, but if you know of something, please let me know.

Also, I'm wondering if the following message is an error or "OK". Normally http 404 code is an error but this entry also says "OK" so I'm confused.

11/15/2022, 13:24:49 [pg3] error: ISY Response: [Try: 1] [myISYid] :: [404 - OK] :: 32.008351ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/SPEED/2.7720000000000002/32

 

[bpwwer]

404 is not found.  So the request was handled by the ISY but the ISY says that node/driver doesn't exist.

You're likely seeing this because something earlier failed to configure the node properly on the ISY.

I don't know of any tools as I said, my networking knowledge is just basic consumer level.  I just know that what you're seeing isn't something that can be fixed in the PG3 software, it's something external to the program that's causing all the issues.

[johnnyt]

 

Ok, I've reported this to UDI. 

 

30 minutes ago, bpwwer said:

404 is not found.  So the request was handled by the ISY but the ISY says that node/driver doesn't exist.

You're likely seeing this because something earlier failed to configure the node properly on the ISY.

So the messages are for this NS and I see the nodes that are reported as not there in ISY so it's strange.

How do I force a reload of the nodes? Restarting NS doesn't do it. I also tried "Load Profile" but it just sits there "clicked" and does nothing. see screenshot.

[bpwwer]

Typically when a node server starts it does an update of all its nodes on the ISY.  It makes will add any nodes that are missing and it will update the drivers to match what the node server thinks they should be.  That's all automatic.

However, in your case, PG3 isn't able to communicate reliably with the ISY so some things are properly added and somethings aren't.

Because of this you're going to see more problems with node servers that have more nodes and more complex nodes than you will with simple node servers. 

[johnnyt]

So I'm working with UDI support on possibility of ISY or Polisy network interface issue.

As part of fathering more data I restarted the WeatherFlow NS with PG3 log level at debug and I'm seeing things that to me suggest issues that may not be related to network, including unhandled errors, "REPORT THIS" messages, and a ton of these:

11/16/2022, 09:33:53 [pg3] debug: 207296 is LOCKED by addnode(), waiting...

I would also mention that it's not like I don't ever see normal response times from ISY. There are plenty of responses that are well below 100ms when I'm not restarting a NS or PG3. I've attached a chunk of logs from yesterday when ISY was obviously not busy to show this.

I guess I'm wondering if every error is really a network problem or is there are other things going on that would continue to happen even if I got all new hardware? 

Or is there anything in the debug logs that would help UDI troubleshoot the problem? I've already sent a bunch of stuff and don't want to send more at this point unless it would be helpful.

Thanks.

PG3Debug-WeatherFlowNSrestart.zip

[bpwwer]

I'm not sure how many more ways I can say this.  PG3 is not getting proper responses back from the ISY.  There's nothing PG3 can do to make the ISY respond or respond in a timely manner.  

In the log you attached, there are many cases where it's taking 10+ seconds before it finally errors out waiting.  I also see cases where it's taking almost that long on the third try to get a response.    

Requests to the ISY that should take less than 100ms are taking 100 times that long and then many are just outright failing.  That's why you see so many node locked, waiting messages.  That's exactly what PG3 is doing, waiting and waiting and waiting. 

When you see an unhandled exception / unhandled rejection it meas that something very unexpected happened, likely a compounding of error responses to the point that it doesn't know what to do.

[johnnyt]

Any idea why the same three nodes or subnodes (not sure what they're called), namely SPEED, GV4, and GUST, are always involved in 404 errors? It's not like those calls don't succeed sometimes - I do see the values change in ISY. It's also not like there are no other nodes that get a 404. it's just that when there are 404's these three nodes are always involved from what I've seen.

11/18/2022, 02:55:13 [pg3] error: No code in error response
11/18/2022, 02:55:13 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 348.399805ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/SPEED/0.5760000000000001/32
11/18/2022, 02:55:13 [pg3] error: No code in error response
11/18/2022, 02:55:13 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 68.046758ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GV4/0.5760000000000001/32
11/18/2022, 02:55:13 [pg3] error: No code in error response
11/18/2022, 02:55:13 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 22.453674ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GUST/0.5760000000000001/32
11/18/2022, 02:56:03 [pg3] error: No code in error response
11/18/2022, 02:56:03 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 36.875933ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/SPEED/0.14400000000000002/32
11/18/2022, 02:56:03 [pg3] error: No code in error response
11/18/2022, 02:56:03 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 16.920775ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GUST/0.14400000000000002/32
11/18/2022, 02:56:03 [pg3] error: No code in error response
11/18/2022, 02:56:03 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 56.705772ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GV4/0.14400000000000002/32
11/18/2022, 02:57:02 [pg3] error: No code in error response
11/18/2022, 02:57:02 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 33.619906ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/SPEED/0.036000000000000004/32
11/18/2022, 02:57:02 [pg3] error: No code in error response
11/18/2022, 02:57:02 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 17.546125ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GUST/0.036000000000000004/32
11/18/2022, 02:57:02 [pg3] error: No code in error response
11/18/2022, 02:57:02 [pg3] error: ISY Response: [Try: 1] [00:21:b9:02:55:cd] :: [404 - OK] :: 57.084195ms - http://192.168.200.251:80/rest/ns/6/nodes/n006_207296/report/status/GV4/0.036000000000000004/32

 

Also, am really struggling to understanding why I get these errors when response times are fine. I get your point that an overloaded (or defective) ISY is bound to cause problems but these problems are happening without an obvious overload.

Attached is PG3 log for today (up to now). As you'll see this NS (#6) results in the most errors. While for sure it is the one that gets the most activity and therefore opportunity for errors, my Airthings NS (#8) updates every 5 minutes, Envisalink-DSC every 6.5 mins or so, and OpenWeatherMap every 10 mins or so. There are a few but not that many Airthings errors and I didn't see any errors from the other ones.

pg3-Nov18.log

[bpwwer]

A 404 error from the ISY means not found.  As to why it's returning that, I have no idea.   

I would think that if it really wasn't found it would never update on the admin console.  But I don't know anything about the ISY software.