Strange new errors and then crash

[johnnyt]

I recently upgraded to PG3.1.14 and, today, after noticing a value in ISY was not getting updated, I check NS logs and it had stopped working at 02:03AM after a number of occurrences of errors I haven't seen before.

2022-11-17 01:48:36,971 Thread-693 udi_interface      ERROR    udi_interface:write: Exception in thread
2022-11-17 01:48:36,973 Thread-693 udi_interface      ERROR    udi_interface:write: Thread-693
2022-11-17 01:48:36,973 Thread-693 udi_interface      ERROR    udi_interface:write: :
2022-11-17 01:48:36,974 Thread-693 udi_interface      ERROR    udi_interface:write: Traceback (most recent call last):
2022-11-17 01:48:36,975 Thread-693 udi_interface      ERROR    udi_interface:write: File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
2022-11-17 01:48:36,979 Thread-693 udi_interface      ERROR    udi_interface:write: self.run()
2022-11-17 01:48:36,980 Thread-693 udi_interface      ERROR    udi_interface:write: File "/usr/local/lib/python3.9/threading.py", line 917, in run
2022-11-17 01:48:36,983 Thread-693 udi_interface      ERROR    udi_interface:write: self._target(*self._args, **self._kwargs)
2022-11-17 01:48:36,984 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Controller.py", line 149, in handler_poll
2022-11-17 01:48:36,986 Thread-693 udi_interface      ERROR    udi_interface:write: self.shortPoll()
2022-11-17 01:48:36,987 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Controller.py", line 153, in shortPoll
2022-11-17 01:48:36,988 Thread-693 udi_interface      ERROR    udi_interface:write: self._query_all()
2022-11-17 01:48:36,989 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Controller.py", line 183, in _query_all
2022-11-17 01:48:36,990 Thread-693 udi_interface      ERROR    udi_interface:write: node.shortPoll()
2022-11-17 01:48:36,991 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Sensor.py", line 104, in shortPoll
2022-11-17 01:48:36,993 Thread-693 udi_interface      ERROR    udi_interface:write: self.query(force=False,authorize=False)
2022-11-17 01:48:36,993 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Sensor.py", line 113, in query
2022-11-17 01:48:36,995 Thread-693 udi_interface      ERROR    udi_interface:write: st = self.controller.api_get(f"devices/{self.serial}/latest-samples")
2022-11-17 01:48:36,996 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/nodes/Controller.py", line 202, in api_get
2022-11-17 01:48:36,997 Thread-693 udi_interface      ERROR    udi_interface:write: res = self.session.get(
2022-11-17 01:48:36,998 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/pgSession.py", line 65, in get
2022-11-17 01:48:37,000 Thread-693 udi_interface      ERROR    udi_interface:write: return(self.response(response,'get'))
2022-11-17 01:48:37,000 Thread-693 udi_interface      ERROR    udi_interface:write: File "/var/polyglot/pg3/ns/<my ISY ID>_8/pgSession.py", line 88, in response
2022-11-17 01:48:37,002 Thread-693 udi_interface      ERROR    udi_interface:write: self.logger.error("Forbidden: %s: text: %s" % (response.url,json_data['message']) )
2022-11-17 01:48:37,002 Thread-693 udi_interface      ERROR    udi_interface:write: KeyError
2022-11-17 01:48:37,003 Thread-693 udi_interface      ERROR    udi_interface:write: :
2022-11-17 01:48:37,004 Thread-693 udi_interface      ERROR    udi_interface:write: 'message'

 

The node value that I noticed not getting updated was GV4 for device s_2930027997. There may be others too - I didn't go fishing for others.

A restart of the NS seems to have fixed things for now.

Attached is log for today, including entries after the restart. If needed I could SSH into Polisy to pull out older log files. Let me know if that would help at all.

Thanks.

 

[Jimbo.Automates]

Thanks for the report @johnnyt, Issue added: https://github.com/UniversalDevicesInc-PG3/udi-poly-Airthings-Consumer/issues/15

The NS received a HTTP 403 "Forbidden" error from the Airthings server and was trying to print the error message, which didn't look like it typically does.  Will need to fix the code to handle this case.

 

[johnnyt]

Thanks. I did notice there were some airthings cloud issues yesterday. After this is fixed and the cloud returns 403, am I right to think it will change the NoderServer Status to "Unconnected" in ISY? And/or would it change the Server Status? I'm a bit confused about the difference and what events change them. Sometimes they are blank for me for a while - I think after an ISY restart.

 

Also do you have timeline for when you will provide a new version? It looks like there are 9 open issues going back to July, most of which do affect my setup. Not critical, of course, although this latest one, while hopefully a rare event, did result in the NS crashing without any notification, i.e. my program notifying me when NS Status is "unconnected" did not trigger.

Thanks again. I really like the ability to get my Airthings data into ISY.

[Jimbo.Automates]

I added info about the server status, hopefully this clears it up: https://github.com/UniversalDevicesInc-PG3/udi-poly-Airthings-Consumer/blob/main/README.md#controller

Sorry, I can't really have a timeline, it all depends on my real job, and what is going in in my life.  I try to keep up on enhancements as much as I can.  Typically I will work on those node servers which are used by more users, and sadly Airthings has the least number of users, so I never get to it, but that doesn't mean it will never happen.  But with 12 node servers in production I get a lot of requests.  But, that being said, a crash is always highest priority, so that will make it this a priority, and when I start working on one, I like to clean up as many issues as possible while I'm in it.

 

[johnnyt]

Thanks for info on different meaning for controller variables/sub nodes (not sure what you call those)

Note that Server Status asks for a number when used in a program, not true/false. I assume '0' means false and anything else means true but would be better if it said true/false (or adjust documentation) 

 

[Jimbo.Automates]

Zero is False, One is True. I'll add that to the list of bugs,
https://github.com/UniversalDevicesInc-PG3/udi-poly-Airthings-Consumer/issues/16

Sent from my Pixel 6 Pro using Tapatalk

Edited November 19, 2022 by Jimbo.Automates

[Jimbo.Automates]

Fixed this and most other issues: 

 

[johnnyt]

Thanks for the update!

Did a restart of the Node Server after seeing message that a restart was needed but it still says v 0.0.6. This is *after* PG3 upgrade to 3.1.16, which I think was supposed to fix this issue with another NS:

 

 

[bpwwer]

I just realized what the problem is with auto-upgrades.

When PG3 3.1.x was released, it included the ability to support multiple different versions of the same node server in the store.  Prior to this, there could only be 1 version of the node server available.  This change caused a change in the format that the version information is maintained.  While PG3 3.1.x is able to work with node servers installed by prior versions those versions don't have the information available to know which version was installed when comparing against the new node server store format.

The version of Airthings that you have installed was 0.0.6 and that used the old style version information.   Version 1.0.0 of Airthings uses the new style version information where there could potentially be other versions available too.  Because of this difference, PG3 doesn't really know if version 1.0.0 is an upgrade to 0.0.6.  

I've been working with the new style version info for so long now that I don't have any test cases like this.  But now that I understand what is happening I can probably tweak the logic to handle this better.

In the mean time, the easy solution is to just re-install the node server to the same slot.  That will update the node server on your system to the new style version.

[johnnyt]

8 hours ago, bpwwer said:

In the mean time, the easy solution is to just re-install the node server to the same slot.  That will update the node server on your system to the new style version.

this is not an "easy solution". I just did uninstall/reinstall of another NS and my nodes have all disappeared in ISY and its hosed all my programs depending on those nodes (6 per device X 8 devices)

 

[Jimbo.Automates]

Do not uninstall then install again, you can re-install in the same slot from the node server store, or by going to the purchases tab.  Sorry Bob did not make that clear enough.

 

[johnnyt]

10 hours ago, Jimbo.Automates said:

Do not uninstall then install again, you can re-install in the same slot from the node server store, or by going to the purchases tab.  Sorry Bob did not make that clear enough.

 

For future reference by qualifying that the re-install needed to be 

Quote

to the same slot

implied having to select the slot at install time. The only time one can physically select a specific slot when installing a NS is when there is nothing in that slot. Otherwise the slot is not shown in the drop down list.

 

 

[Jimbo.Automates]

implied having to select the slot at install time. The only time one can physically select a specific slot when installing a NS is when there is nothing in that slot. Otherwise the slot is not shown in the drop down list.
 
 

There should be a button that says reinstall in slot #. Where # is the current slot. If not, then you are on a pretty old version of PG3.

Sent from my Pixel 6 Pro using Tapatalk

[bpwwer]

I used "same slot" because you could have a situation like this:

where the node server was installed in more than one slot so you have to pick the correct slot to re-install from (4 or 15 in the example above).

[johnnyt]

51 minutes ago, Jimbo.Automates said:

There should be a button that says reinstall in slot #. Where # is the current slot. If not, then you are on a pretty old version of PG3.

Yes there's a reinstall button but given the following preamble that suggested 0.0.6 was really old and missing basic stuff that would even allow PG3 and/or the NS to react properly to a "normal" reinstall using the button you mention it, plus the comment that the "easy solution is to just re-install the node server to the same slot. " suggested to me that a "fresh" install was needed.

Quote

 While PG3 3.1.x is able to work with node servers installed by prior versions those versions don't have the information available to know which version was installed when comparing against the new node server store format.

The version of Airthings that you have installed was 0.0.6 and that used the old style version information.   Version 1.0.0 of Airthings uses the new style version information where there could potentially be other versions available too.  Because of this difference, PG3 doesn't really know if version 1.0.0 is an upgrade to 0.0.6.  

My bad. Won't happen again. In my defense, you and Bob have likely done more installs, reinstalls, and uninstalls than you even remember so it's all pretty obvious to you. Next time I do this (using reinstall button) it will be my first time doing it.

 

[Jimbo.Automates]

No worries, that's why I apologized for the process not being fully explained.

Sent from my Pixel 6 Pro using Tapatalk