At first all my servers showed green with nodes.  I had to stop each of them and start or restart. Did it with the log tab selected for each node & you can be sure it started. That got all my servers going again. They only came up with this manual intervention and not a eisy or pg3x reboot. 

2 hours ago, sjenkins said:

 

At first all my servers showed green with nodes.  I had to stop each of them and start or restart. Did it with the log tab selected for each node & you can be sure it started. That got all my servers going again. They only came up with this manual intervention and not a eisy or pg3x reboot. 

I had this same behavior after update and on additional restarts of eisy.

After restarting PG3x or rebooting the eisy I always need to do a start on each node server as they come up disconnected.

eisy 5.5.4

PG3x  3.1.20

My Node Servers come to a stop for no reason that I am aware of.  Need than to start back up one at a time. I captured the log when I think it happened. I changed it to debug so maybe better info if it happens again. So far seems to be daily since updating to 3.1.20.

/20/2023, 13:35:23 [pg3] error: uncaughtException: read ECONNRESET
Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
1/20/2023, 13:35:23 [pg3] error: uncaught ECONNRESET exception:  Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
1/20/2023, 13:35:23 [pg3] info: Caught SIGTERM/SIGINT Shutting down.
1/20/2023, 13:35:23 [pg3] info: Stopping running node servers...
1/20/2023, 13:35:23 [pg3] info: [WeatherLink(1)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [VenstarCT(2)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [WirelessTag(3)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [Notification(4)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [Wemo(5)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [WeatherBit(6)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [LiFX(7)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [iTach-IR(8)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [iTach-IR(9)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [Bond(10)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [TeslaEV(12)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [AVRemote(13)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [ELK(14)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: [iTach-IR(17)]: Stopping Node Server
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns
1/20/2023, 13:35:23 [pg3] info: Making reqeust to socket for /rest/pg3.stop.ns

Edited January 20, 20233 yr by garybixler

Same issue.  After a reboot I need to manually restart each node on PG3.

The node server's not starting on restart is a bug.  I'm looking into it.

@garybixlerYour log shows that some type of network error occurred that PG3 was unable to handle, so it shutdown.  it doesn't provide any information on what may have caused the error or what PG3 was trying to do when the error occurred.

Was there anything in the log prior to the exception that may help?

Is it happening at the same time every day? 

It seems to be happing around the same time everyday. Early afternoon. I'll look for any previous problem before the shutdowns. Thanks.

Pretty much this over and over before the servers were stopped. If it happens tomorrow I'll get a debug log.

 

1/20/2023, 13:08:37 [pg3] info: [00:21:b9:02:60:38_14] controller reporting command DOF
1/20/2023, 13:09:39 [pg3] info: [00:21:b9:02:60:38_3] wtcontroller reporting command DON
1/20/2023, 13:09:52 [pg3] info: [00:21:b9:02:60:38_4] controller reporting command DON
1/20/2023, 13:11:09 [pg3] info: [00:21:b9:02:60:38_12] controller reporting command DOF
1/20/2023, 13:12:37 [pg3] info: [00:21:b9:02:60:38_14] controller reporting command DON
1/20/2023, 13:12:44 [pg3] info: Verifying node servers are installed on IoX correctly
1/20/2023, 13:12:44 [pg3] info: IoX entry for WeatherLink::1 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for VenstarCT::2 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for WirelessTag::3 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for Notification::4 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for Wemo::5 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for WeatherBit::6 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for LiFX::7 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for iTach-IR::8 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for iTach-IR::9 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for Bond::10 OK
1/20/2023, 13:12:44 [pg3] info: IoX entry for TeslaEV::12 OK
1/20/2023, 13:12:45 [pg3] info: IoX entry for AVRemote::13 OK
1/20/2023, 13:12:45 [pg3] info: IoX entry for ELK::14 OK
1/20/2023, 13:12:45 [pg3] info: IoX entry for iTach-IR::17 OK
1/20/2023, 13:13:47 [pg3] info: [00:21:b9:02:60:38_14] zone_30 reporting command DON
1/20/2023, 13:14:04 [pg3] info: [00:21:b9:02:60:38_14] zone_30 reporting command DON
1/20/2023, 13:16:09 [pg3] info: [00:21:b9:02:60:38_12] controller reporting command DON
1/20/2023, 13:16:37 [pg3] info: [00:21:b9:02:60:38_14] controller reporting command DOF
1/20/2023, 13:17:44 [pg3] info: Verifying node servers are installed on IoX correctly
1/20/2023, 13:17:44 [pg3] info: IoX entry for WeatherLink::1 OK
1/20/2023, 13:17:44 [pg3] info: IoX entry for VenstarCT::2 OK

@garybixlerBased on that info, PG3x is trying to verify the node servers installed on the IoX, it does that every 5 minutes. While it was doing that, the IoX device refused the connection, causing the exception. 

I'll see if I can trap that exception since that shouldn't cause PG3x to shutdown.

But since it's happening around the same time every day, is there something happening on the IoX at that time that would make it too busy to accept connections from PG3x?

 

@bpwwerLast night it broke from the pattern and happened around 10:30:00 pm instead of early afternoon. At that time there would be nothing happening except receiving data from the NSs. I'll look for any correlations.

Thanks

On 1/19/2023 at 3:02 PM, glarsen said:

Seeing the same thing with ST-Sonos (same for ST-NuHeat).  PG3x 3.1.20 also. No node server log, not much in the pg3x log (attached): 

 

@bpwwer, any thoughts on why @auger66 and I can't get the ST-Sonos node server to start properly?  I am experiencing the same issue with St-NuHeat.  (Reported earlier in topic - log snippet attached).  Node.js issues?

Thanks!

@glarsenThe node servers that are written in node.js need to be updated to work with PG3x. I've updated ST-Inventory but haven't had time to update the others yet.

1 hour ago, bpwwer said:

The node servers that are written in node.js need to be updated to work with PG3x.

Does this also apply to the LifX ns ?

14 hours ago, asbril said:

Does this also apply to the LifX ns ?

No.  @xKing just needs to fix the requirements file so it uses the working version of the library it needs. 

I was having issues with LifX so decided to upgrade to 5.5.4 \ 3.1.20 yesterday morning.  I now have the red screen problem in PG3 so I'll issue a ticket.  I've tried all the regular fixes - update packages again, power cycle, flush cache, different PC, etc.  It appears the ISY UUID can't be found or manually added and related to auth.ns

 

Jan 21 10:12:15 eisy daemon[69112]: 1/21/2023, 10:12:15 [pg3] ^[[31merror^[[39m: Getting IoP UUID failed: Error: connect ECONNREFUSED 127.0.0.1:8080
Jan 21 10:12:15 eisy daemon[69112]: 1/21/2023, 10:12:15 [pg3] ^[[31merror^[[39m: Is the IoX service enabled?  The IoX service must be enabaled for PG3 to start.
Jan 21 10:12:15 eisy daemon[69112]: 1/21/2023, 10:12:15 [pg3] ^[[32minfo^[[39m: PG3 shutdown complete with code: 0
Jan 21 10:12:31 eisy daemon[69112]: 1/21/2023, 10:12:31 [pg3] ^[[32minfo^[[39m: Starting PG3 version 3.1.20

Jan 21 11:03:48 eisy daemon[2441]: 1/21/2023, 11:03:48 [pg3] ^[[32minfo^[[39m: Making reqeust to socket for /rest/pg3.auth.ns
Jan 21 11:03:48 eisy daemon[2441]: 1/21/2023, 11:03:48 [pg3] ^[[31merror^[[39m: uncaughtException: socket hang up
Jan 21 11:03:48 eisy kernel: pid 1332 (udx-freebsd-x64), jid 0, uid 0: exited on signal 11 (core dumped)
Jan 21 11:03:48 eisy daemon[2441]: Error: socket hang up
Jan 21 11:03:48 eisy daemon[2441]:     at connResetException (node:internal/errors:711:14)
Jan 21 11:03:48 eisy daemon[2441]:     at Socket.socketOnEnd (node:_http_client:518:23)
Jan 21 11:03:48 eisy daemon[2441]:     at Socket.emit (node:events:525:35)
Jan 21 11:03:48 eisy daemon[2441]:     at endReadableNT (node:internal/streams/readable:1359:12)
Jan 21 11:03:48 eisy daemon[2441]:     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Jan 21 11:03:48 eisy daemon[2441]: 1/21/2023, 11:03:48 [pg3] ^[[31merror^[[39m: uncaught ECONNRESET exception:  Error: socket hang up
Jan 21 11:03:48 eisy daemon[2441]:     at connResetException (node:internal/errors:711:14)
Jan 21 11:03:48 eisy daemon[2441]:     at Socket.socketOnEnd (node:_http_client:518:23)
Jan 21 11:03:48 eisy daemon[2441]:     at Socket.emit (node:events:525:35)
Jan 21 11:03:48 eisy daemon[2441]:     at endReadableNT (node:internal/streams/readable:1359:12)
Jan 21 11:03:48 eisy daemon[2441]:     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Jan 21 11:03:48 eisy daemon[2441]: 1/21/2023, 11:03:48 [pg3] ^[[32minfo^[[39m: Caught SIGTERM/SIGINT Shutting down.
Jan 21 11:03:48 eisy daemon[2441]: 1/21/2023, 11:03:48 [pg3] ^[[32minfo^[[39m: Stopping running node servers...

16 minutes ago, JeffR said:

I was having issues with LifX so decided to upgrade to 5.5.4 \ 3.1.20 yesterday morning.  I now have the red screen problem in PG3 so I'll issue a ticket.  I've tried all the regular fixes - update packages again, power cycle, flush cache, different PC, etc.  It appears the ISY UUID can't be found or manually added and related to auth.ns

 

Jan 21 10:12:15 eisy daemon[69112]: 1/21/2023, 10:12:15 [pg3] ^[[31merror^[[39m: Getting IoP UUID failed: Error: connect ECONNREFUSED 127.0.0.1:8080
Jan 21 10:12:15 eisy daemon[69112]: 1/21/2023, 10:12:15 [pg3] ^[[31merror^[[39m: Is the IoX service enabled?  The IoX service must be enabaled for PG3 to start.

It sound like the IoX service didn't start back up after the upgrade.  PG3x needs to the IoX service to be running or it can't start.

Can you start the admin console from the IoX finder?

1 hour ago, bpwwer said:

It sound like the IoX service didn't start back up after the upgrade.  PG3x needs to the IoX service to be running or it can't start.

Can you start the admin console from the IoX finder?

I had trouble with the ELK NS coming back to life after the upgrade to 5.5.4 yesterday and it appears to be it couldn't connect to the MQTT broker.  I was able to issue it a Start command and it takes off running after that.  I've been informed there is a known bug in PG3 after a full reboot and will need to wait for the next release.  IoX & PG3 is starting up for me.  I'm wondering if the problem could too be related to this full reboot bug?  

1 hour ago, JeffR said:

I've tried all the regular fixes - update packages again, power cycle, flush cache, different PC, etc.

Not sure if you logged out and back in (for PG3x) as part of your regular troubleshooting fixes, but I sometimes find that is needed for things to work properly after I upgrade. 

2 hours ago, bpwwer said:

No.  @xKing just needs to fix the requirements file so it uses the working version of the library it needs. 

This is exactly why I am in favor of annual fees for Node Server developers. I want them to be compensated not only for the initial development, but also for maintenance.

Edited January 22, 20233 yr by asbril

@bpwwerMy NSs are still stopping in the early afternoon and now in the early evening. I can not find any reason for this. The IoX is not doing anything at those times. Checked all the programs. I attached the PG3x log file but it just shows the same as before. I am now only running the essential NSs in hopes this can be narrowed down. It happened today at 1:35 PM and again at 7:07PM. I checked the IoX event viewer and nothing seemed to be a problem. This issue may have started after upgrading to 5.5.4. Can't be sure however. Also removed the latest updated Z-Wave devices.

Node servers.txt

23 hours ago, bpwwer said:

It sound like the IoX service didn't start back up after the upgrade.  PG3x needs to the IoX service to be running or it can't start.

Can you start the admin console from the IoX finder?

Michel determined 5.5.4 has a bug to fix.  My Admin password had a $ in it and once we changed that everything came back online.  It was a very strange issue and even took a while for him to track down but all is good now.

@bpwwer Hi, I upgraded to version 3.1.21 this morning and although I didn't notice any NSs stopping I did find that they did in the PG3x log. But automatically restarted. I attached the PG3x log.

Thanks

Node servers.txt

@bpwwerUpgraded everything yesterday and PG3x running on 3.1.21 and migrated from Polisy to EISY and Restored Polisy PG3 file into EISY. 

I am still trying to review if there are any basic problems after the migration, so I am not yet ready to transfer the Portal and NS licenses.  I noticed that the PG3x showed the Node Servers in the dashboard, but they were all labeled “unmanaged “. 

I assume this is normal since the licenses have not been transferred, is that correct?

No, unmanaged means it thinks something else is controlling the node servers in a specific slot.  Migrating should have updated everything but it sounds like the IoX didn't get updated.

You can do the migration again, it doesn't hurt to re-run it.

The lack of license should prevent the node servers from running but they should be 'managed', not 'unmanaged'.

21 minutes ago, bpwwer said:

No, unmanaged means it thinks something else is controlling the node servers in a specific slot.  Migrating should have updated everything but it sounds like the IoX didn't get updated.

You can do the migration again, it doesn't hurt to re-run it.

The lack of license should prevent the node servers from running but they should be 'managed', not 'unmanaged'.

@bpwwerThank you for resetting me.  I think the problem may have been me... I believe initially I "restored" the PG3 Polisy backup... I missed the "Migrate Backup from PG3" button.  I did the migrate function and it indicated the ISY had to be manually reconfigured, so I did that.  

There were numerous system pop-up messages and after all the system messages finished, the paid node servers went from "unmanaged" to "connected".  Only the Node from the Polisy Portal remained "unmanaged", but as I said, the licenses have not been moved yet for Portal or the node servers.

I tried restarting the node servers and they all gave me the message that there was no valid license... I suspect that means all is OK except for me to actually migrate the license, then the restart should be fine.

I think everything is OK at this point but if there is anything else users need to watch out for after the migration, please comment.