Illusion

Upgrade of OS to 14.1 and IoX to 5.9.1 has corrected the issue. I guess we never get to know what was going on. I am going on the theory that it was something in IoX 5.8.4 as my update to that firmware coincides pretty closely with the start of the issue. Why no one else had the issue, and why we could not figure out what mechanism was the cause will remain a mystery.

Upgrade to wired Polisy went well. Only notes are I did not get the 4 beeps at the end. Did get beeps throughout. Took about 20m PG3x and system no longer show up at https://polisy.local:8443/WEB/sysconfig.txt) and https://polisy.local:3000 I have to use the IP address.

I am going to move away from system startup time research. I am going with @bpwwer view that this is an admin console bug. Reboot/restarts later in the afternoon were still 41s off, but the AM/PM differential went away. Based on the fact that the time displayed in the admin console, that ssh shows Polisy knows the correct time, and that the email I get from ISY on reboot is timestamped correctly in the message body by the ISY, I do not want to spend any more time on this. I did 6 hours of tests on this issue yesterday. I am not opposed to coming back to this, but I would like to focus my efforts on the fact that IoX does not restart if I am not connected to the UD Portal. I think the answer lies in that. Either that or figuring out what is different about stopping internet access for the IoX vs other forms of internet interruption. I think that could give us a great data point. Data point #4: and Variations on those tests #3: @Geddy Regarding your idea of changing networking cable and topology, I thought that would be a good thing to take as far as I could while testing the startup time anomaly extensively. So I strung ethernet cable together to run my Polisy on my neighbor's network. So different WAN, different LAN, different cable, different router. Same results. IoX restarts unless I break the connection to the UD Portal.

Okay, here is what I currently have investigating the time/NTP issue. @Guy Lavoie Thank you for that suggestion. Can confirm that Polisy has correct time as verified through your ssh suggestion. The time is not always so far off. I restored internet at 11.50.30am and system shows: I restored at 12.02.30p and system shows: I restored internet at 12.09.03p and system shows: I am not sure this strangeness has anything to do with my issue at all, but is fascinating. I was wrong about this only happening with the internet drop and restore. There is the same time discrepancy if I reboot the Polisy or 'Restart IoX' from the configuration pane. The same discrepancy shows up if I do a remove/restore power to the Polisy. I had missed this by not paying attention to the AM/PM or because all those test were done in the AM and being off by a few seconds during a reboot is impossible to detect because of system startup. It is off by a few seconds and always seems to show only AM. I will do more testing in the afternoon.

@BigMojo Oh wow! I had not even caught the AM/PM differential. Interesting

@Geddy Just tested with different network cable connected to a switch connected to a mesh node in another part of the house. Same results. @gregkinney 1. Network cable replaced. 2. Tested on completely different power supply. A battery, so no wall outlet or circuit breaker involved at all in this test. Full isolation from home power. Same results. New Data point! I was doing all these test by just pulling the ethernet cable out and plugging it back in since I had taken the effort to get to the back of my Polisy to change/reroute the network cable. Most of my tests are done by shutting off the WAN in the router just because it is the easiest. But what I realized on one of the restarts, the restart time did not seem right. Because it is a physical removal and restoration, I can know the exact second that the internet was restored. Internet was restored at 12.06.15p. IoX System status shows a startup time of 12.05.34!!! IoX is showing a startup time 41s BEFORE the ethernet cable was plugged in! I do not know if it is relevant, but I know computers like correct time. If I select the restart IoX button, or I hard reboot the Polisy, the startup time is correct. (The system time in the Admin Console is correct in all cases)

My Notification plugin will always notify all my devices that have UD Mobile with a message "Notification Node Server Standard Edition Startup." any time the Notification plugin starts up. I do not seem to be able to stop this. Other messages like "PG3 is starting" respond to the "Plugin Notifications (PG3)" group, and messages like "(Polisy) is now offline" respond to the System Notifications group.. But I do not seem to be able to stop the notification from this plug in. It goes to every device that has UD Mobile. Is there something I am missing?

That is a good one! I started to test this as a factor but stopped when I realized that the test where I unplug the ethernet cable preserves WAN IP address. I suspect that a reboot of the router does as well since the modem retains power. But turning off my internet over and over throughout the day creates other problems that I have to limit. So much so that I really wish the 'block internet access' switch would work to cause the issue. But your suggestions does make me think that there is something there. So my Polisy has a reserved IP address. The Polisy is not set to a static IP internally, but the router always gives it the same LAN IP address. But if I 'block internet access' for just this device, the LAN IP address remains constant. But this should also be true if I remove power to the ONT. The LAN never goes down in that case. I am still unable to reconcile the difference between Data points we have so far: #4 and #5 I feel like there is a piece of information here that if I could isolate it might inform people smarter than me.

There is new OS for the Polisy coming out soon. UD and I plan to install that and continue testing, but Michel has low confidence that it will correct this particular issue. We were focused on hardware right up to the point that I can make it stop by severing the connection to the UD Portal. That is where we kinda hit a wall. Unfortunately, I do not have other hardware that I could switch to, and that would be a major endeavor. Our current plan is to install the new OS soon and if the issued continues do some network traffic monitoring. But I started this forum post because people much smarter than me do not think that is going to fix anything.

I have lots of Insteon, a little bit of Z-wave. I have the Zmatter board. Power supply is an APC DC UPS (CP12142LI) https://www.apc.com/us/en/product/CP12142LI/network-ups-12vdc-lithium-battery19500mah-bms-4led/ I have considered altering the power supply for testing, but I cannot bring myself to make the effort because IoX does not reboot unless the portal is connected and I loose and restore internet. I cannot see how a different power supply would alter that variable. TCP/IP is used extensively in my plugins and networking module.

Ooh, that is a good one! I like it. But alas, no change. IoX still reboots. I turned off "System Notifications", "UD Communications", and "Plugin Notifications" for all groups/devices. I also stopped the Notification Plugin for this test. Yes. I kinda have to. I did not use to, because my Polisy and my IoX programs are built to never restart unless I actively and manually do it. I have over 300 nodes, so it is a long time to 'Query All' if I am working on something that requires a reboot or restart. My Polisy is on a DC UPS, so does not loose power unless power is off for a long time. But now that my ISY restarts an average of once a week or so, I have to have 'Query at Restart' enabled or my ISY does not know the status of any of my devices. My whole wake up programs fall apart in this case. Disaster! I really feel like we can move away from the program angle. Disabling ALL programs does not stop the behavior. I admit to being a somewhat poor program writer, but I have been pretty great about never creating any kind of looping program. Surely UD would have seen that in the logs as you suspect. I like your thinking that the connection to a wifi device on the local network is involved, but the only thing I have like that is my Tempest weather station and associated plugin. But I feel like that is excluded by the stopping PG3x test. Further, my ISY does not get the weather info from the internet, it gets it locally from the LAN. And even if it did, I feel like the severing of the connection to the portal causing a change in behavior would exclude that since it has nothing to do with the portal.

Thanks Geddy! In my version of Mac, I think it is Option + Command + R to 'Reload Page From Origin' but that worked. Thank you. On Sequoia Command + Shift + R is 'Show Reader' Command + R is indeed 'Reload Page'

@RRalstonLogs have been captured and sent to UD. If your theory were correct, what is the relevant difference between having my IoX connected to the portal vs not connected to the UD Portal? Would not the electrical status change be the same either way, so we should see the same results in those two scenarios?

@Guy Lavoie Interesting. I had not, but now I have. Will answer here and update the original post to include data point. Disabling all programs does not stop IoX from restarting on loss and restoration of internet.

I have an open ticket with Universal Devices. I have been troubleshooting and narrowing down this issue I am having for close on 6 months now. The UD ticket was opened once I closed in on cause and effect enough to have something useful to say. Ticked has been actively addressed back and forth for the past month and now we are stuck, with no ideas how to address nor further narrow down the cause. So I come to the forum. If my Polisy looses internet access, when it is restored the IoX component will restart. Data points we have so far: If I unplug the ethernet cable to the Polisy, when I plug it back in the IoX will restart. If I reboot the router, when the router comes back up, IoX will restart. If I unplug power to the ONT (fiber optical network terminal), when I power the ONT back on, IoX will restart. This most closely simulates the issue that needs to be corrected. Any time my ISP goes down for a moment, when it is restored my IoX will restart. Maintenance by Frontier. in the middle of the night by the service results in an IoX restart. I have an ASUS router. It has a button with each device connected to disable access to the internet for that device. If I throw that switch in the router, my IoX shows that the portal is offline, the UD portal shows that my IoX is offline. But the IoX DOES NOT RESTART when I once again allow access from the Polisy to the internet. My ASUS router also allows me to stop the WAN with a radio button. If I do that everything on my network looses access to the internet, but the LAN is maintained and devices on the network can still connect to each other. Re-enabling the WAN on the router will result in in an IoX restart. I do not know what the difference between killing the WAN and killing internet access for just the Polisy is. I do not understand the results of #5 vs #4. The network has to be down for 30 seconds or more to cause the behavior. The Polisy does not produce any beeps or lights when the IoX restarts after internet restoration. Variations on those tests: Stopping PG3x does not stop IoX from restarting on loss and restoration of internet access. So I think we can exclude interaction from the few plugins I have. Disabling all programs does not stop IoX from restarting on loss and restoration of internet access. If I sever the connection between IoX and the UD Portal, IoX DOES NOT RESTART with loss and restoration of the internet. Parameters we are dealing with: The Polisy is up to date The Iox is version 5.8.4 The Polisy is wired network only. No WiFi The only inbound connection to the system I can think of is the UD Portal. The test are 100% repeatable with results always the same. Polisy, Polyglot are not restarting/rebooting during the event. UD cannot recreate the issue on their end. It is tough to troubleshoot remotely because it requires the loss of network connection to cause the issue, so tough for them to watch happen. Questions: Has anyone else experienced this? Does anyone have any suggestions on possible cause and or corrective action? Does anyone have any other tests that could give us additional data to provide more clarity on what is happening?