nodes are not starting on reboot

[bchats]

after a power failure or a clean reboot, the polisy comes back up but none fo the nodes in v3 auto start, they all always remain in a stopped state, any way to change this?

[DennisC]

33 minutes ago, bchats said:

after a power failure or a clean reboot, the polisy comes back up but none fo the nodes in v3 auto start, they all always remain in a stopped state, any way to change this?

This is a bug that UD is working on correcting. I had some conversations on this with them yesterday. For now, on a restart, I have been manually querying the nodes.

[bpwwer]

3 hours ago, bchats said:

after a power failure or a clean reboot, the polisy comes back up but none fo the nodes in v3 auto start, they all always remain in a stopped state, any way to change this?

Which platform and what version of PG3?

The latest versions are:

eisy: PG3x 3.1.22

polisy: PG3 3.1.17

The initial release of PG3x for eisy had a bug that would prevent all node servers from starting but that has been fixed in the later releases.

[DennisC]

Bob, I'm on the releases you listed and experience the same issues. I opened a support ticket with UD yesterday thinking it was related to the random reboots I was experiencing.

I have waited 15 minutes for node servers to populate the admin console. In my case, the node servers are running, but don't populate until queried.

The node servers I experience this with is Weatherflow, Notification, Wireless Tags, and sometimes TimeData.

Weatherflow doesn't have a query button associated with it on the node page,  but you can right click on the name in the left hand tree and query from there.

[bpwwer]

@DennisCI've seen your ticket, your problem seems different as the node servers are starting.

I'm trying to reproduce and does "reboot" mean a system reboot or an IoX reboot?

[DennisC]

16 minutes ago, bpwwer said:

@DennisCI've seen your ticket, your problem seems different as the node servers are starting.

I'm trying to reproduce and does "reboot" mean a system reboot or an IoX reboot?

Reboot is a restart of eisy.

And, yes, in my case the node servers are starting and running fine.

I opened the ticket with UD because I thought it might have been rated to the eisy random reboots, which I am told there will be a fix out today, or the false upgrade packages available notice.

[bpwwer]

28 minutes ago, DennisC said:

Reboot is a restart of eisy.

And, yes, in my case the node servers are starting and running fine.

I opened the ticket with UD because I thought it might have been rated to the eisy random reboots, which I am told there will be a fix out today, or the false upgrade packages available notice.

Hmm, that's not what Michel said when I asked him what reboot meant, he claimed it was a restart of the isy service. 

I've been trying both and so far I've not seen any issues with WeatherFlow reporting data or the admin console displaying the data after either type of reboot.

It is possible to trace the flow of data from the node server to the IoX using the  PG3(x) log files.  The log levels needs to be set to debug to do this.

The node server log will report every time the node servers sends an update to PG3(x)

The PG3(x) log will report every time it sends an update to IoX and will also show if it was successfully sent or not.

The more node servers that are running the more difficult it is to trace, but since WeatherFlow should be reporting data every 60 seconds it's not too bad:

For example, WeatherFlow node server reports

2023-02-17 15:59:08,766 MQTT       udi_interface.interface INFO     interface:_message: Successfully set 240018 :: WINDDIR to 2 UOM 76

then the PG3 log shows

2/17/2023, 15:59:08 [pg3] debug: ISY Response: [Try: 1] [00:21:b9:02:61:f1] :: [200] :: 0.935454ms http://127.0.0.1:8080/rest/ns/1/nodes/n001_240018/report/status/WINDDIR/2/76
2/17/2023, 15:59:08 [pg3] debug: MQTT Results: [ns/status/00:21:b9:02:61:f1,1] :: {"set"[{"address":240018,"driver":"WINDDIR","value":"2","uom":76}]}
2/17/2023, 15:59:08 [pg3] debug: [PUBLISH: udi/pg3/ns/clients/00:21:b9:02:61:f1_1] {"set":[{"address":240018,"driver":"WINDDIR","value":"2","uom":76}]}

The first line says it sent it to the ISY and was successful (status 200). The second line is the results back to the node server, and the third line is it actually sending the results to the node server.

 

[DennisC]

10 minutes ago, bpwwer said:

Hmm, that's not what Michel said when I asked him what reboot meant, he claimed it was a restart of the isy service. 

I've been trying both and so far I've not seen any issues with WeatherFlow reporting data or the admin console displaying the data after either type of reboot.

It is possible to trace the flow of data from the node server to the IoX using the  PG3(x) log files.  The log levels needs to be set to debug to do this.

The node server log will report every time the node servers sends an update to PG3(x)

The PG3(x) log will report every time it sends an update to IoX and will also show if it was successfully sent or not.

The more node servers that are running the more difficult it is to trace, but since WeatherFlow should be reporting data every 60 seconds it's not too bad:

For example, WeatherFlow node server reports

2023-02-17 15:59:08,766 MQTT       udi_interface.interface INFO     interface:_message: Successfully set 240018 :: WINDDIR to 2 UOM 76

then the PG3 log shows

2/17/2023, 15:59:08 [pg3] debug: ISY Response: [Try: 1] [00:21:b9:02:61:f1] :: [200] :: 0.935454ms http://127.0.0.1:8080/rest/ns/1/nodes/n001_240018/report/status/WINDDIR/2/76
2/17/2023, 15:59:08 [pg3] debug: MQTT Results: [ns/status/00:21:b9:02:61:f1,1] :: {"set"[{"address":240018,"driver":"WINDDIR","value":"2","uom":76}]}
2/17/2023, 15:59:08 [pg3] debug: [PUBLISH: udi/pg3/ns/clients/00:21:b9:02:61:f1_1] {"set":[{"address":240018,"driver":"WINDDIR","value":"2","uom":76}]}

The first line says it sent it to the ISY and was successful (status 200). The second line is the results back to the node server, and the third line is it actually sending the results to the node server.

 

Well, we never specifically spoke about it until he asked me what reboot meant. Wouldn't those items restart with an eisy reboot?

Other than ssh in to eisy and issuing a UDX restart command, I wouldn't know how to restart those services.

If there is something specific you want me to test, I can test it tomorrow morning. I'm out of the house now.

[bchats]

i'm on PG3 Version 3.1.16   looking now it looks like there's an upgrade to 3.1.18 is it worth doing the upgrade?

[DennisC]

@bpwwer

I received a request from Michel to install v5.5.7. After taking my backup, I set the node server logs to debug and proceeded to upgrade packages on eisy. 

I had to do the upgrade packages twice to get the admin console to load with some status'. None of the node servers had data in the admin console after about 20 minutes. When I checked on PG3x, none of the node servers were running and attempts to start them failed. I restarted PG3x and all of the node servers started up.

When I restarted the admin console, Insteon and Zwave had loaded. It took about 7 minutes for all of the node servers to populate the admin console, however, this is the first restart of eisy since upgrading to v5.5.6 that all of the node servers populated the admin console.

I'm wondering if all my issues were related. Anyway, I will try a few more restarts of eisy tomorrow, as I don't want to push my luck further tonight.

[bpwwer]

16 hours ago, bchats said:

i'm on PG3 Version 3.1.16   looking now it looks like there's an upgrade to 3.1.18 is it worth doing the upgrade?

3.1.18 hasn't been released to production yet so if 3.1.16 is working then no PG3 upgrade is needed at this time.

[bpwwer]

15 hours ago, DennisC said:

@bpwwer

When I restarted the admin console, Insteon and Zwave had loaded. It took about 7 minutes for all of the node servers to populate the admin console, however, this is the first restart of eisy since upgrading to v5.5.6 that all of the node servers populated the admin console.

I'm wondering if all my issues were related. Anyway, I will try a few more restarts of eisy tomorrow, as I don't want to push my luck further tonight.

Possibly, PG3x is more dependent on IoX and UDX so issues with those can impact PG3x.   If you do see the issue with the nodes not populating try the following:

1. Set one of the node servers that isn't populating to debug log (if WeatherFlow is one, it's a good one to check) and monitor.  If it's sending updates to PG3x it should be obvious.  If not, you'll likely see very little log activity.

2. If you see it sending to PG3x, open the PG3x log and set it to debug.  You should see the http requests to IoX with status 200, meaning it was able to send the update to the IoX.  If not, copy a couple of entries and post here.

3. Open the event viewer on admin console and set to level 3. You should see related entries for each request sent by PG3x.

Based on what the results are for each step, we can narrow down where the problem is happening.

[DennisC]

1 minute ago, bpwwer said:

Possibly, PG3x is more dependent on IoX and UDX so issues with those can impact PG3x.   If you do see the issue with the nodes not populating try the following:

1. Set one of the node servers that isn't populating to debug log (if WeatherFlow is one, it's a good one to check) and monitor.  If it's sending updates to PG3x it should be obvious.  If not, you'll likely see very little log activity.

2. If you see it sending to PG3x, open the PG3x log and set it to debug.  You should see the http requests to IoX with status 200, meaning it was able to send the update to the IoX.  If not, copy a couple of entries and post here.

3. Open the event viewer on admin console and set to level 3. You should see related entries for each request sent by PG3x.

Based on what the results are for each step, we can narrow down where the problem is happening.

Will do. 

I am going to experiment this afternoon with one or two reboots of eisy to check. Any advantage to setting node servers on debug before the reboot, or will that generate too much traffic?

[bpwwer]

Probably not.  The node server log doesn't really show historic log entries (unless you download it) and the PG3x log won't maintain that level, it will revert to info on restart.

[DennisC]

1 hour ago, bpwwer said:

Possibly, PG3x is more dependent on IoX and UDX so issues with those can impact PG3x.   If you do see the issue with the nodes not populating try the following:

1. Set one of the node servers that isn't populating to debug log (if WeatherFlow is one, it's a good one to check) and monitor.  If it's sending updates to PG3x it should be obvious.  If not, you'll likely see very little log activity.

2. If you see it sending to PG3x, open the PG3x log and set it to debug.  You should see the http requests to IoX with status 200, meaning it was able to send the update to the IoX.  If not, copy a couple of entries and post here.

3. Open the event viewer on admin console and set to level 3. You should see related entries for each request sent by PG3x.

Based on what the results are for each step, we can narrow down where the problem is happening.

I rebooted a few times and all of the node servers loaded. They take a little time, about 7 minutes, but they all loaded with a query.

[bpwwer]

Do  you mean they all populated after 7 minutes or that you had to manually query them to get them to populate?

In general, you shouldn't need to manually query to get them to populate. However, because of how node servers work and timing, it is possible that some initial values could be lost on startup and that the node server could take quite a while before it sends updates for some values.

For example, WeatherFlow will send all of the weather values when it starts.  If it sends those before the IoX is ready  to receive them, it won't re-try until it has a new value to send.  For things like wind speed, that could be in 60 seconds, but temperature could be 5 or 10 minutes.  So it would slowly populate those values over time.

For some of the weather service node servers, it can be 30 minutes before it gets an update.  It's going to depend on how often the node server sends updated values.

[DennisC]

19 minutes ago, bpwwer said:

Do  you mean they all populated after 7 minutes or that you had to manually query them to get them to populate?

In general, you shouldn't need to manually query to get them to populate. However, because of how node servers work and timing, it is possible that some initial values could be lost on startup and that the node server could take quite a while before it sends updates for some values.

For example, WeatherFlow will send all of the weather values when it starts.  If it sends those before the IoX is ready  to receive them, it won't re-try until it has a new value to send.  For things like wind speed, that could be in 60 seconds, but temperature could be 5 or 10 minutes.  So it would slowly populate those values over time.

For some of the weather service node servers, it can be 30 minutes before it gets an update.  It's going to depend on how often the node server sends updated values.

Prior to upgrading to v5.5.7, I had to manually query them to get them to populate since on eisy after a reboot of eisy.

After upgrading to v5.5.7, yes, all nodes populated on their own, but it took about 7 minutes. 

Thanks for the explanation, I understand.

[GFrazier]

Went from 5.5.5 to 5.5.7 have Zmatter board in Polisy.    Zmatter is spotty on devices supported, however now I can't get Polyglot v3 started.    Followed directions, multiple reboots.   I can get to Polyglot v2.   Please help - day 4 of no nodes servers on Polyglot v3 last known version was 3.1.16.  

 

After multiple reboots and sudo service udx restart.... looked as though modules were stuck.

Edited February 22, 2023 by GFrazier