whywork Posted November 12, 2023 Share Posted November 12, 2023 Bah and Ugh! Following migration to PG3x two daya ago, I get constant Node/App up down messages and PG3 starting messages. Background: Polisy Pro 5.7 via ethernet with PG2 and PG3 running for months with no Polyglot errors/messages. End of life - updated to PG3x two days ago After several reboots - PG3x 3.2.16 finally seemed up and running ALL - Old installed node servers were up and running. BUT Got 37 messages in UD mobile last night about all installed node servers starting and stopping. Also got several messages about PG3 starting. Tried soft reboot, Cold reboot and "update packages" - No Joy. Now when I try to logon to PG3x, get http error auth failure then try again immediately and do get to logon to PG3x but some node servers are disconnected. (see PG3x log snippet below) ALSO - from java IoX launcher - still get offered ability to logon to PG2 (yes PG2) - it does go nowhere but still it is offered. Bah Went from working PG2 and PG3 to buggy PG3x installation Suggestions? 11/12/2023, 24:00:45 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:00:45 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:00:47 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/GV3/4/45 11/12/2023, 24:00:47 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 324.423465ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/GV3/4/45 11/12/2023, 24:00:47 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:00:48 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:00:49 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_645646/report/status/GV3/4/45 11/12/2023, 24:00:49 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 289.859629ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_645646/report/status/GV3/4/45 11/12/2023, 24:00:50 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:00:51 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:00:52 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_645954/report/status/GV3/4/45 11/12/2023, 24:00:52 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 187.748774ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_645954/report/status/GV3/4/45 11/12/2023, 24:00:53 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:00:54 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:00:54 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_646253/report/status/GV3/4/45 11/12/2023, 24:00:55 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 208.306427ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_646253/report/status/GV3/4/45 11/12/2023, 24:00:56 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:00:56 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:00:58 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_842242/report/status/GV3/4/45 11/12/2023, 24:00:58 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 186.951931ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_842242/report/status/GV3/4/45 11/12/2023, 24:00:59 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:00 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:00 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1207309/report/status/GV3/4/45 11/12/2023, 24:01:00 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 168.404506ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1207309/report/status/GV3/4/45 11/12/2023, 24:01:02 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:02 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:02 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1448163/report/status/GV3/4/45 11/12/2023, 24:01:03 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 19.262084ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1448163/report/status/GV3/4/45 11/12/2023, 24:01:16 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:17 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:18 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:18 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:19 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:19 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:19 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/CLIHUM/77/22 11/12/2023, 24:01:20 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 358.756243ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/CLIHUM/77/22 11/12/2023, 24:01:20 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/DEWPT/26/17 11/12/2023, 24:01:21 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/GV3/0/45 11/12/2023, 24:01:21 [pg3] info: Received commands on topic udi/pg3/ns/status/00:21:b9:00:f1:ee_1: set 11/12/2023, 24:01:21 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 429.390843ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/DEWPT/26/17 11/12/2023, 24:01:21 [pg3] info: IoX Response: [Try: 1] [00:0d:b9:53:36:9c] :: [200] :: 204.519976ms - http://127.0.0.1:8080/rest/ns/1/nodes/n001_1212003/report/status/GV3/0/45 11/12/2023, 24:01:22 [pg3] info: Received commands on topic udi/pg3/ns/status/00:0d:b9:53:36:9c_1: set 11/12/2023, 24:01:23 [pg3] info: IoX Request: [Try: 1] [00:0d:b9:53:36:9c] :: - http://127.0.0.1:8080/rest/ns/1/nodes/n001_645646/report/status/GV3/0/45 11/12/2023, 24:01:23 [pg3] info: IoX Response: [Try: 1] Link to comment
tlightne Posted November 12, 2023 Share Posted November 12, 2023 PG3x is rock solid so did you pull power on Polisy/Eisy after installing? If not pull power wait 20 - 30 seconds then reaply power and wait for reboot. User id and password for PG3x are the same as what you use to login the Polisy/Eisy. I would open a ticket and let UD support check it out. Link to comment
whywork Posted November 13, 2023 Author Share Posted November 13, 2023 99 Node failed then Node connected messages today Package upgrade, warm boot, cold boot. Started looking at uptime and noticed what look like average load averages increases here is the output of "top" last pid: 6508; load averages: 2.30, 2.15, 2.19 up 0+08:26:27 11:54:48 62 processes: 2 running, 60 sleeping CPU: 52.2% user, 0.0% nice, 1.8% system, 0.2% interrupt, 45.9% idle Mem: 249M Active, 681M Inact, 50M Laundry, 1212M Wired, 1728M Free ARC: 674M Total, 127M MFU, 323M MRU, 643K Anon, 6209K Header, 210M Other 308M Compressed, 751M Uncompressed, 2.43:1 Ratio Swap: 4132M Total, 4132M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 3198 polyglot 11 101 0 4534M 219M CPU1 1 372:11 101.80% node 5396 000db95336 5 52 0 109M 68M usem 2 167:42 99.42% python3. 5356 0021b900f1 508 52 0 394M 251M usem 2 181:49 2.46% python3. CPU at 52% Yikes Link to comment
bpwwer Posted November 13, 2023 Share Posted November 13, 2023 4 minutes ago, whywork said: 99 Node failed then Node connected messages today Package upgrade, warm boot, cold boot. Started looking at uptime and noticed what look like average load averages increases here is the output of "top" PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 3198 polyglot 11 101 0 4534M 219M CPU1 1 372:11 101.80% node 5396 000db95336 5 52 0 109M 68M usem 2 167:42 99.42% python3. 5356 0021b900f1 508 52 0 394M 251M usem 2 181:49 2.46% python3. What is the plug-in that I highlighted in bold? That plug-in seems to be what is causing all the CPU usage. Also, it's username seems wrong. The username should be the Polisy's uuid_slot for the plug-in. Top only displays part of the username, but all plug-ins should have the same uuid with just the slot part different. For the two you show, the uuid portion is very different. Link to comment
whywork Posted November 13, 2023 Author Share Posted November 13, 2023 Hi, Thanks for the quick reply. I appreciate your help. I have no clue as to who/what 000db95336 user is or the plugin it is related to. This is a "plain vanilla" Polisy Pro that was running perfectly OK with PG2 and PG3 three days ago with assorted "plug-ins"/node servers. CPU utilization spike came only after following posted instructions to migrate to PG3x. I am unsure as to how to query "top" to learn more about the "rogue" user/process. I am happy to help with further information if I can. Suggestions? Link to comment
Solution whywork Posted November 15, 2023 Author Solution Share Posted November 15, 2023 Some closure I have a PolisyPro in production and a retired/disconnected Isy-994i pro (polisy Pro connected to USB dongle 2448A7 - which means I hope to never have PLM power supply capacitor issues - with Isy-994i pro pre-configured as backup, but I digress). With PG2 and PG3 running - both PolisyPro and Isy-994i pro were configured/had node servers assigned. PG2 and PG3 connected to the PolisyPro but happily "ignored" the missing Isy-994i pro. After the migration with just PG3x running it seems PG3x is VERY upset that the Isy-994i pro is missing! Why do I say that? I deleted Isy-994i pro from PG3x configuration and the FreeBSD CPU usage went back down to 3% (from 50% - 70% or greater) Everything started working and no more up/down starting messages. Seems like everything is working OK even after 24hrs. Perhaps PG3x could be a little more forgiving; my wife tells me I should try to be. Link to comment
Recommended Posts