Support thread for: PG3x 3.1.21 (January 23, 2023)

[bpwwer]

Once you migrate the licenses, it should be good.

[bigDvette]

Not sure if this just started with 3.1.21 or if it is even a node server issue but suspect it is.

my occupancy node server just disappears.  I have to go to portal and go to occupancy Node Server configuration and click configure Node Server/. Then I have to restart pg3x on eisy and then it works for a while.  
 

its cold here and it keeps setting my thermostats back based on occupancy and pissing off my wife

[bigDvette]

On 1/30/2023 at 8:54 PM, bigDvette said:

Not sure if this just started with 3.1.21 or if it is even a node server issue but suspect it is.

my occupancy node server just disappears.  I have to go to portal and go to occupancy Node Server configuration and click configure Node Server/. Then I have to restart pg3x on eisy and then it works for a while.  
 

its cold here and it keeps setting my thermostats back based on occupancy and pissing off my wife

@bpwwerhave you seen anyone else with this problem. Trying to decide if I need to report it somewhere else.

[bpwwer]

@bigDvetteI added it to my list of things to do.  It sounds like it could be a bug in the node server verification code, but I have't had a chance to look at it yet.

[bpwwer]

On 2/1/2023 at 1:34 PM, bpwwer said:

@bigDvetteI added it to my list of things to do.  It sounds like it could be a bug in the node server verification code, but I have't had a chance to look at it yet.

I installed and did some testing with the Occupancy node server and I'm not seeing any issues.   There is at least on support ticket open that sounds like it might be a similar issue.

PG3x shouldn't have any interaction with the Occupancy node server other than to show an Unmanaged dashboard entry for it.  I've verified that PG3x isn't attempting to delete or modify it in any way.

When you say it disappears, where does it disappear from?  From the admin consoles node server list?  Or just from the PG3x's dashboard?  or both?  

Is there something that triggers this?  I.E. when you restart (IoX, PG3x, the eisy) or is it random?

 

[DennisC]

@bpwwer

I migrated yesterday from Polisy to eisy with ZMatter Zwave. I managed to get everything working fine. This morning, I had trouble getting in to Polyglot, so I rebooted eisy.

Everything went down hill from there. After the start up, none of the Node Servers were running. I tried several time to start them manually with no luck. After several restart PG3's, reboots of eisy, and removal of power from eisy, I got 4 out of my 6 running. The 2 that wouldn't run were Weatherflow & Elk.

After trying to stop and start these two node servers several times, I tried rebooting eisy one more time and after that I was able to manually start Elk Node Server. Weatherflow would still not start.

Of interest, is the Weatherflow log would also not populate. I tried setting logging to debug and tried to start up several times more.

As a last resort, I tried re-installing Weatherflow in to the same slot, like I had to do yesterday. THe Node Server apparently updated from v3.0.25 to 3.0.26 (the last time I checked for updates was just before the migration). I tried several times to stop and start the Node Server, but Weatherflow will still not run.

I have pm'ed you the log files.

Thanks for any help you can provide.

dc

[bpwwer]

@DennisCThis looks like the same issue we've been trying to debug for the past couple of days.  One (or two) node servers don't start and won't start manually with a "already starting" response.

Last night I managed to track the issue to the service that PG3x uses to start node servers.  PG3x calls the service to start the node server and .... nothing, crickets.  the service never starts the node server and never responds back to PG3x so PG3x is sitting there waiting for it.  

It's seems to be somewhat random as which node server(s) are effected and it can change when PG3x is restarted.  For some folks it's been the YoLink node server, for others the Elk, For me, it switches between YoLink and WeatherFlow.  I don't have access to the source for that service so right now I'm waiting to here back from the person that does.

[DennisC]

4 minutes ago, bpwwer said:

@DennisCThis looks like the same issue we've been trying to debug for the past couple of days.  One (or two) node servers don't start and won't start manually with a "already starting" response.

Last night I managed to track the issue to the service that PG3x uses to start node servers.  PG3x calls the service to start the node server and .... nothing, crickets.  the service never starts the node server and never responds back to PG3x so PG3x is sitting there waiting for it.  

It's seems to be somewhat random as which node server(s) are effected and it can change when PG3x is restarted.  For some folks it's been the YoLink node server, for others the Elk, For me, it switches between YoLink and WeatherFlow.  I don't have access to the source for that service so right now I'm waiting to here back from the person that does.

Do you think additional tries to restart might help or hurt? Is it worth trying to reboot again or just restart PG3?

Right now, the Elk Node Server is running and it is the one I don't want to be without.

[bpwwer]

While trying to track it down I was restarting my PG3x very frequently and it would tend to be the same node server that it would get stuck on for a while.  Initially it was YoLink, then it switched to WeatherFlow, then it switched back to YoLink.  

So I'd say there's a 50/50 chance that restarting PG3x will result in the Elk node server not starting.

[DennisC]

1 minute ago, bpwwer said:

While trying to track it down I was restarting my PG3x very frequently and it would tend to be the same node server that it would get stuck on for a while.  Initially it was YoLink, then it switched to WeatherFlow, then it switched back to YoLink.  

So I'd say there's a 50/50 chance that restarting PG3x will result in the Elk node server not starting.

Thanks, I will sit tight. If you need anything else or for me to test something, please let me know.

[JTsao]

This is happening to me - I (unfortunately) am trying to migrate from my Polisy to eISY and appear to have random node servers displaying the "already starting" issue, with the node server in slot 1 = DSC Envisalink, always exhibiting this problem (and slot 2 as well)...

[DennisC]

8 hours ago, bpwwer said:

@DennisCThis looks like the same issue we've been trying to debug for the past couple of days.  One (or two) node servers don't start and won't start manually with a "already starting" response.

Last night I managed to track the issue to the service that PG3x uses to start node servers.  PG3x calls the service to start the node server and .... nothing, crickets.  the service never starts the node server and never responds back to PG3x so PG3x is sitting there waiting for it.  

It's seems to be somewhat random as which node server(s) are effected and it can change when PG3x is restarted.  For some folks it's been the YoLink node server, for others the Elk, For me, it switches between YoLink and WeatherFlow.  I don't have access to the source for that service so right now I'm waiting to here back from the person that does.

@bpwwer

Not positive this is related, but I believe it is, since this started this morning, none of my Notifications have worked using the Notification Node Server.

While some of the missed Notifications are for weather and the Weatherflow Node Server is not online, some were for the Elk Node Server. Both the Elk and Notification Node Servers indicate they are online.

When right clicking on individual Notification programs and selecting run then, the program indicates it ran, but no notification is sent to Pushover.

Other network resources appear to be working.

All were working before having to reboot eisy this morning because of difficulty getting in to PG3x.

[sjenkins]

10 hours ago, DennisC said:

@bpwwer

Not positive this is related, but I believe it is, since this started this morning, none of my Notifications have worked using the Notification Node Server.

 

Have you tried stopping & restarting the node server. There is a bug currently they are working on where the node server shows running but is not. Happens for me in a number of servers usually indicated in the admin where the server node does not indicate running. 

[DennisC]

1 minute ago, sjenkins said:

Have you tried stopping & restarting the node server. There is a bug currently they are working on where the node server shows running but is not. Happens for me in a number of servers usually indicated in the admin where the server node does not indicate running. 

Multiple times, read my post above Bob's response for the fu story.

[sjenkins]

5 minutes ago, DennisC said:

Multiple times, read my post above Bob's response for the fu story.

Sorry Dennis, didn’t read up the screen. I actually had the same issue with YoLink as you & worked with Bob on it day before last. I was describing the other NS issue which resolved the way I described. 

[DennisC]

40 minutes ago, sjenkins said:

Sorry Dennis, didn’t read up the screen. I actually had the same issue with YoLink as you & worked with Bob on it day before last. I was describing the other NS issue which resolved the way I described. 

No worries, we are all in this together.

[DennisC]

12 hours ago, DennisC said:

@bpwwer

Not positive this is related, but I believe it is, since this started this morning, none of my Notifications have worked using the Notification Node Server.

While some of the missed Notifications are for weather and the Weatherflow Node Server is not online, some were for the Elk Node Server. Both the Elk and Notification Node Servers indicate they are online.

When right clicking on individual Notification programs and selecting run then, the program indicates it ran, but no notification is sent to Pushover.

Other network resources appear to be working.

All were working before having to reboot eisy this morning because of difficulty getting in to PG3x.

I was able to resolve the issue of not receiving notifications from the node server. It turns out, after all was working on Friday and early Saturday morning, some how the host IP addresses in the network resources reset to the old Polisy IP. Once they were updated to the IP for eisy, notifications resumed.

I have no idea how this happened, as all was working on Friday.

[DennisC]

23 hours ago, bpwwer said:

@DennisCThis looks like the same issue we've been trying to debug for the past couple of days.  One (or two) node servers don't start and won't start manually with a "already starting" response.

Last night I managed to track the issue to the service that PG3x uses to start node servers.  PG3x calls the service to start the node server and .... nothing, crickets.  the service never starts the node server and never responds back to PG3x so PG3x is sitting there waiting for it.  

It's seems to be somewhat random as which node server(s) are effected and it can change when PG3x is restarted.  For some folks it's been the YoLink node server, for others the Elk, For me, it switches between YoLink and WeatherFlow.  I don't have access to the source for that service so right now I'm waiting to here back from the person that does.

@bpwwer,

I'm trying to improve my ability to read error logs. Hope you don't mind my asking:

Is this the error you're seeing when WeatherFlow attempts to start:

[pg3] info: Starting log stream for pg3frontend_yZlDB :: /var/polyglot/pg3/ns/0021b90261f0_1/logs/debug.log

2/5/2023, 12:12:46 [pg3] debug: MQTT Results: [frontend/ns/dcrupi] :: {"startLogging":{"success":true}}

2/5/2023, 12:12:46 [pg3] debug: [PUBLISH: udi/pg3/ns/clients/00:21:b9:02:61:f0_5] {"shortPoll":{}}

2/5/2023, 12:12:51 [pg3] debug: Sending logfile to frontend :: /var/polyglot/pg3/logs/pg3-current.log

2/5/2023, 12:12:51 [pg3] warn: stopLogging: unwatch failed, TypeError: Cannot read properties of undefined (reading 'pg3frontend_yZlDB')

[bpwwer]

@DennisCNo, I don't mind answering questions at all.

The warning your seeing is normal under some circumstances.  When the UI asks PG3x for a log, PG3x has to set up a process to watch the log file and send any new entries to the UI.  When the UI switches away from viewing a log file, it sends a "stopLogging" messages to PG3x telling it to stop that watch process on the log file.

There are cases where the UI might not send the "stopLogging" message so to make sure we don't leave watch processes running when we don't have to, we have the UI send that message at other times, just to be sure. 

In this case, it sent the "stopLogging" to be sure the process isn't running, before it attempts to start watching the log file.  However, since it wasn't watching the logfile, PG3x traps the error (UI, you told me to stop watching something I'm not watching) and outputs the warning.

 

There is no error in the log for node server not starting issue.  The log will show a couple of messages indicating that the it's attempting to start the node server, but doesn't make it past a certain point.  It's waiting for a response from the start service, but the service never responds. 

PG3x is written in such a way that this doesn't block anything else from happening so everything else should continue working normally.  You just end up with one or two node servers in a waiting to start state.

[DennisC]

11 minutes ago, bpwwer said:

@DennisCNo, I don't mind answering questions at all.

The warning your seeing is normal under some circumstances.  When the UI asks PG3x for a log, PG3x has to set up a process to watch the log file and send any new entries to the UI.  When the UI switches away from viewing a log file, it sends a "stopLogging" messages to PG3x telling it to stop that watch process on the log file.

There are cases where the UI might not send the "stopLogging" message so to make sure we don't leave watch processes running when we don't have to, we have the UI send that message at other times, just to be sure. 

In this case, it sent the "stopLogging" to be sure the process isn't running, before it attempts to start watching the log file.  However, since it wasn't watching the logfile, PG3x traps the error (UI, you told me to stop watching something I'm not watching) and outputs the warning.

 

There is no error in the log for node server not starting issue.  The log will show a couple of messages indicating that the it's attempting to start the node server, but doesn't make it past a certain point.  It's waiting for a response from the start service, but the service never responds. 

PG3x is written in such a way that this doesn't block anything else from happening so everything else should continue working normally.  You just end up with one or two node servers in a waiting to start state.

Got it, thanks very much.

[Defiant Computing]

The folder structure seems to differ from the PG2 version.  Does anyone know where I'm supposed to put my devlist.yml file?

 

**Never mind, I figured out it is configured in the devlist paramater

Edited February 15, 2023 by Defiant Computing

[garybixler]

I have a couple of devices that offer MQTT integration and was wondering if they could be added to PG3x? One is Genmon that has an MQTT addon.

Thanks

 

[bpwwer]

@garybixlerI'm not sure what you're asking.

PG3(x) uses MQTT for communication between the PG3(x) components, not for communication with external devices.  A node server is needed to create a communication bridge to external devices.

[bigDvette]

On 2/3/2023 at 11:59 AM, bpwwer said:

I installed and did some testing with the Occupancy node server and I'm not seeing any issues.   There is at least on support ticket open that sounds like it might be a similar issue.

PG3x shouldn't have any interaction with the Occupancy node server other than to show an Unmanaged dashboard entry for it.  I've verified that PG3x isn't attempting to delete or modify it in any way.

When you say it disappears, where does it disappear from?  From the admin consoles node server list?  Or just from the PG3x's dashboard?  or both?  

Is there something that triggers this?  I.E. when you restart (IoX, PG3x, the eisy) or is it random?

 

So still happening:  happened today.  When it happens it disappears from pg3x.  If I go to portal it doesn’t even show it is installed.  I click install and it installs back in the same slot 6 out of 14.  The it reboots my ISY and it is back/

 

I do think it happens when I reboot or lose connectivity.  It happened sometime last night and of course then my house thinks I’m not home and I wake up and it is like 60 in house:  I haven’t turned off that stuff because I wouldn’t know it isn’t working. 
 

oh and the node is missing in ISY console.  If I restart pg3 it does not come back.  Only way to get back is to go to portal and click install:  

[tazman]

7 hours ago, bigDvette said:

So still happening:  happened today.  When it happens it disappears from pg3x.  If I go to portal it doesn’t even show it is installed.  I click install and it installs back in the same slot 6 out of 14.  The it reboots my ISY and it is back/

 

I do think it happens when I reboot or lose connectivity.  It happened sometime last night and of course then my house thinks I’m not home and I wake up and it is like 60 in house:  I haven’t turned off that stuff because I wouldn’t know it isn’t working. 
 

oh and the node is missing in ISY console.  If I restart pg3 it does not come back.  Only way to get back is to go to portal and click install:  

Have you updated? I was having the occupancy not working after reboot but it has been fixed for me since IoX 5.5.6