Z-wave devices stop working after PG3 update

[tlightne]

I am also having issues after performing the upgrade package yesterday. Many of my ZWave devices no longer work. I have reloaded Polisy from a previous backup with all my PG3 node servers stopped and all my devices are working. If I start PG3 node servers several of the Zwave devices just stop working. Reloaded Polisy once again with backup and all Zwave is back and has been available for a couple of hours. I have not started and PG# nodesrvers. Not sure where to go next.

[Geddy]

@tlightne your issue didn't seem to be same as where you posted it so split it out for more visibility for your issue.

Will you please confirm your ISY Version (Firmware and UI) and Z-Wave controller?

 

[asbril]

I have no idea whether it is related or not, but yesterday (for the second time)  my Zwave dongle stopped working. Like the previous time, several weeks ago, it took several reboots, probably at least 5, to get the dongle to respond again.

My Polisy is inside a closet and there is no reason to think that the dongle was moved, so I must assume that it is some kind of bug or setting. Next time it happens (hopefully never) I will try to remember to download the error log and send it to UD.

[tlightne]

ISY 5.4.4 UI 5.4.4 Zwave USB Zooz 700. Polisy has been running for last 2 hours fine. Again I have not started any node servers. Everything was running fine with PG# 3.6. I pushed Upgrade Packages from AC and after restarting PG3 version 3.10 issue started.

 

Edited September 24, 2022 by tlightne

[tlightne]

Attached is the ISY Error Log but can not upload the PG3 lpg3_9-24-2022_64227_PM.zipog getting an error.

ISY Error Log.v5.4.4__Sat 2022.09.24 08.17.35 PM.txt

[gviliunas]

@tlightneI think I am having the same root problem. I initially didn't recognize this as a Z-wave not working problem as I have only one Z-wave switch but many Z-Wave sensors (that I didn't notice as not working). For me, I noticed that none of my programs would run after a reboot. Initially this was intermittent and "playing" with PG3 NS start-stop would fix the issue and IOP would take-off and run. 

In an initial ticket to UD, Michel wrote that my programs were hung because IOP could not communicate with a device(s).

Now, I noticed that when programs hang after a reboot, if I unplug the 700 dongle, IOP immediately queries the Insteon devices, runs programs and and all NS data seems to be displayed properly. All PG3 NS are enabled and seem to always disconnect / connect properly with each reboot. Hmmm seems to be related to the Z-Wave to IOP interface and not so much to PG3 (but I don't know for sure). Unfortunately IOP does not support hot-plugging the 700 dongle so I didn't expect Z-wave to begin working after plugging the dongle back in.

I tried replacing the 700 dongle (with power off) with another good 700 dongle and restoring a previous backup with no change in symptoms.

I am able to control my one light switch via the original dongle connected to my PC using the Silicon Labs tool so I don't think that the 700 dongles are the problem. The Polisy FreeBSD usbconfig command shows a "Silicon Labs Zooz ZST10700 Z-Wave Stick) connect so I don't think it's a usb problem.

I have updated my original ticket and am awaiting UD's analysis.

One way to possibly confirm if we are seeing the same problem is to ask if your programs will run when you have problems with your Z-wave? I created a simple Turn on light, Wait 5 sec, Turn off light program. In my case, when I do a Run Then, The icon turns green (forever) but no light on or off.

[tlightne]

@gviliunas My programs with a zwave device does not run when I have my problem. I have several zwave switches, window shades, and sensors. Not all the zwave devices fail. Example= I have three zwave shades, one fails the others work fine. Out of 5 zwave switches two fail. two out of 8 sensors fail. My Polisy has been running fine for close to 20 hours now with out and of my PG3 node servers running so I do not think it has anything to do with the zwave dongle. 

[tazman]

@gviliunasI have the same problem as you and it came down to @Michel Kohanimnot being able to help me figure it out and he said to send the items to UDI and they would figure out what was wrong but with the upcoming release of the new equipment I did not want to go without the hardware so I just decided to try not to reboot and live with it for now. It sounds like @tlightne has a different problem since some of his Z-Wave stuff is working and it does not sound like the whole system hangs until Z-Wave dongle becomes responsive.

[gviliunas]

Thanks @tazman and @tlightne for your feedback! I missed the fact that only part of tlightne's z-wave devices weren't working. Yes, definitely a different problem. Maybe time for me to try re-imaging to rule out the hardware.

 

[bpwwer]

@tlightneYour ISY log is filled with -170001 errors.  That seems to correspond to the internal queue being full.  That would make sense if something was blocked/stuck and could cause all the symptoms described.

Unfortunately, almost anything could cause this.

PG3 does do a lot of communication with the ISY/IoP when it starts.  So it is possible that something it sends, or even the amount that it is sending, on startup triggers something to block.   One test would be to disable (stop) all of the node servers and then reboot.  PG3 will remember the last state of the node servers so it should start with all node servers disconnected (not running).  Then slowly start them manually one by one giving it a minute or two to stabilize between each.   If it works, that tells us something.  If it starts showing the same symptoms after one specific node server is started, that tells us something.

I suspect that your PG3 log is just too large to attach here.  The log is reset a midnight each day so the earlier in the day you are able to do any testing, the shorter the log will be and you'll have a better chance of it attaching here.  If you still have problems, you can PM me and I'll send you my email address and we can see if email can handle the size.

[tlightne]

@bpwwer I rebooted yesterday with all the node servers stopped IOP ran with no problems about 20 hours before I started Ambient Weather Node server that has ran for a couple of hours. After seeing your response I will wait until morning and start more node servers hopefully the logs will be small enough to attach.

Thank you

[MWareman]

During the last 'pkg upgrade' - the following got updated:

Sep 23 20:33:05 polisy pkg[2395]: python39 upgraded: 3.9.13 -> 3.9.14

Since then, I see this error when most PG3 Polys start (as well as having similar issues identified in this thread - high latency on zwave devices for example):

9/25/2022, 11:06:13 [pg3] error: [LiFX(6)] :: STDERR: ld-elf.so.1: /lib/libc.so.7: version FBSD_1.7 required by /usr/local/lib/libpython3.9.so.1.0 not found

I have a feeling that a missing Python dependency like this might affect a lot more than just polys running. libpython3.9.so.1.0 was updated from the UDI package server - but libc.so.7 should likely have also been updated to be conformant is it does not appear this was the case from the logs. 

[tlightne]

@bpwwerI have three node servers started this morning and all is working as designed so far. Ambient Weather, Open Weather and Notification Node server. These are what I consider my priority node servers. I have to shift gears as we are under a hurricane watch, I need to do some prep work around the house will be evacuating Tuesday afternoon so it maybe a few days before I can get back to this problem.  

Thank you

[bpwwer]

@tlightneTake care and I hope you make it through without any damage.   Keep safe.

[brians]

I have had my Z-Wave Zooz dongle seem to become unresponsive a few times since I switched over to Polisy this past January. I attribute it to just a glitch and hard reboot and has fixed each time.

The last time this happened however, I did notice that a Zooz ZEN25 double plug was causing the issue... sending out too much Z-Wave traffic so I unplugged and disabled and everything is fine. It is of note that I have had issues with this plug before and the last Zooz Z-Wave stick update seemed to have resolved it, but guess not.

I'm not worrying about it am waiting for new ZMatter board.

 

[asbril]

Just now for the 3rd time the Zwave dongle stopped responding and obviously none of my Zwave devices were working. As in the previous instances, after some 5 reboots of IoP, the system came back. I have no idea what it can be and I sent copies of the log and error log to UD Support, hoping that they can identify the issue.

If the Zwave dongle would be deficient, could I simply buy a new one and perform a Restore of  my latest backup ?

[tlightne]

@bpwwerI am finally back from Hurricane Ian all is good. I have started all my node servers and everything (Zwave) seems to be working as designed. I do have an Issue with PG2. I keep a couple of node servers just for backup purposes. When I go into the PG2 dashboard the two node servers do not show as managed. I know PG2 is no longer supported but thought you may have a suggestion. Thank you..

 

[MWareman]

1 hour ago, tlightne said:

@bpwwerI am finally back from Hurricane Ian all is good. I have started all my node servers and everything (Zwave) seems to be working as designed. I do have an Issue with PG2. I keep a couple of node servers just for backup purposes. When I go into the PG2 dashboard the two node servers do not show as managed. I know PG2 is no longer supported but thought you may have a suggestion. Thank you..

 

That is how node servers managed by PG3 show in the PG2 console. Are you sure these are not PG3 node servers?

[bpwwer]

When something shows up as unmanaged, that means that the node server is installed on the ISY/IoP but the version of Polyglot that you are running does not see that node in it's database so it assumes some other version of Polyglot is managing it.

PG2 has no way to fix the database when this happens other than by restoring from a backup.

[tlightne]

57 minutes ago, MWareman said:

That is how node servers managed by PG3 show in the PG2 console. Are you sure these are not PG3 node servers?

Thanks I understand. Most of my node servers are PG3 but two are PG2 and they all show as unmanaged. 

[tlightne]

43 minutes ago, bpwwer said:

When something shows up as unmanaged, that means that the node server is installed on the ISY/IoP but the version of Polyglot that you are running does not see that node in it's database so it assumes some other version of Polyglot is managing it.

PG2 has no way to fix the database when this happens other than by restoring from a backup.

Can I uninstall and them reinstall them in the same slots in the ISY??

[Geddy]

38 minutes ago, tlightne said:

Can I uninstall and them reinstall them in the same slots in the ISY??

Before you do that try running the update process again from Admin Console 

(Photo credit to @MrBill)

I had a similar situation where PG2 node servers were showing stopped in PG2 after the initial 3.1.6 was released. I think it was either something in a later release that fixed up some of that or the process didn't complete as expected on the initial run. 

Just also make sure there's nothing in PG3 also trying to write to slots 1 & 2 since PG2 is using those slots. 

Finally...aren't these also available in PG3, why not just use the PG3 version of these two? 

[bpwwer]

2 hours ago, tlightne said:

Can I uninstall and them reinstall them in the same slots in the ISY??

yes, that should work.  But you'll have to manually remove them from the ISY slots because PG2 isn't managing them at this point, it can't remove them.  Once you delete them from the ISY (node servers -> configuration -> slot1, click delete), PG2 should show the slots as free in roughly 5 minutes, then you can reinstall.

[wmcneil]

On 9/28/2022 at 11:26 PM, asbril said:

Just now for the 3rd time the Zwave dongle stopped responding and obviously none of my Zwave devices were working. As in the previous instances, after some 5 reboots of IoP, the system came back. I have no idea what it can be and I sent copies of the log and error log to UD Support, hoping that they can identify the issue.

If the Zwave dongle would be deficient, could I simply buy a new one and perform a Restore of  my latest backup ?

@asbril, did you ever get resolution of your problem? I am experiencing a complete loss of ZWAVE communication after having upgraded to PG3 3.1.12 . I have tried power several soft reboots, and a power cycle reboot of the polisy box to no avail. 

[bpwwer]

2 hours ago, wmcneil said:

@asbril, did you ever get resolution of your problem? I am experiencing a complete loss of ZWAVE communication after having upgraded to PG3 3.1.12 . I have tried power several soft reboots, and a power cycle reboot of the polisy box to no avail. 

You should submit a ticket.

Unfortunately with the current "upgrade packages" you get all pending updates, not just PG3.  It's highly unlikely that upgrading PG3 would effect z-wave.  But it is very possible that some other update would.