Random ISY reboot

[MrBill]

My normally stable ISY rebooted this morning, never had that happen before except when i screwed something up. I was in the yard, when i got the notification, the admin console wasn't open.  I count uptime hours and that variable was at 1349 (or around 56 days). 

Checking the error log at the time of reboot I find:

Version 5.0.13D

Anyone have any idea what might have occurred?    I'm getting ready to be gone for an extended period and need to deal with anything quickly.

Thanks.

 

 

[Techman]

The admin console isn't designed to be left permanently open. It's possible that you encountered a (lack of) memory issue.

[MrBill]

5 minutes ago, Techman said:

The admin console isn't designed to be left permanently open. It's possible that you encountered a (lack of) memory issue.

 

2 hours ago, MrBill said:

..... the admin console wasn't open....

 

Specifically I never leave the admin console just open.  Only if i'm working in it.   Previously the only time I have ever had reboots that were caused by some mistake I made. 

Not the case today.  I added a few more Insteon modules and added a program in the last week, but nothing major nor complex.. and the system has been running stable for 56 days, the point of my post is to attempt to figure out the root cause of the random reboot because if there is an issue I need to resolve it before being away for nearly two weeks in the near future.  I have absolutely no idea why this would have happened, that's why I came here to seek help, sorry that I was unclear.

[Techman]

The log you posted shows the Queue as FULL.  Do you have programs running that could be consuming memory?  You may also want to update to the latest firmware release which may contain bug fixes.

[MrBill]

(Sorry for the delayed reply here, we had a family emergency and our grand kids were with us for four days unexpectedly.  The forum also didn't email me that there had been a new reply.)

 

I don't know what queue would be full. I don't know what queue to watch, or how to watch it.   I don't even know what is queued.    I really have no idea what the errors in the error log mean, again this is why I came and posted about the incident here.  It made zero sense to me, it still makes zero sense to me.  Literally nothing was happening at that time.  No scheduled events since Sunrise.  No one was in the house turning anything on or off that would trigger a program.   The only programs that I have that routinely show "running" are the heartbeat monitoring programs for door sensors batteries and the program that adds an hour to the uptime counter.  Everything else shows "idle" 99.9% of the time, programs pretty much run because of a trigger and exits quickly and becomes "Idle" again. 

As for updating to 5.0.14, I think at this point I'll have to wait until we return.  I don't want to introduce something new within a week of leaving the country.  It  was stable at 56 days since the last reboot before this unexpected reboot event.    I was surprised that the admin console doesn't indicate a release after the one I'm on, I realize it's beta but still I'm surprised the update notification functionality is not in use here.   That is once I moved to beta 5.x that new 5.x beta release would show in the admin console the way 4x general releases told me when to upgrade.  I didn't really want to upgrade to 5.x when it was alpha, but once it went to beta I did for exactly one need, otherwise I could have stayed on the current stable general release. 

[Techman]

In the Admin Console Click on Tools | Diagnostics | System Status.  That will show your ISY available memory.  If it's low then that could be an issue.  It's possible your variables or notifications are consuming your memory.

I would also try updating your firmware to the current beta version. The 5.x.x is a work in progress and you might have come across a bug.

Being that you're running a beta version you won't be notified of new beta updates. Update notifications are only sent out for final release firmware. You'll need to check the forum for the most recent alfa or beta release.

 

[MrBill]

Thanks. 

As of now it looks good to me.  I will watch this and see if it remains basically unchanged, which it should.  Or builds up over time due to some type of issue or memory leak.

 

 

Anything else I should be watching for.

[MrBill]

The regular log leading up to to the random reboot doesn't give any clues either.

 

Door Switches / LowerSlider		        Status	On	Sat 2019/03/16 11:01:10 AM	System	Log
Door Switches / LowerSlider		        Status	Off	Sat 2019/03/16 11:01:13 AM	System	Log
OUT Upper Deck Cans+# / {hide}Spa Motor	Status	Off	Sat 2019/03/16 11:02:31 AM	System	Log
OUT Lower Slider OH+# / {hide}Spa Motor	Status	Off	Sat 2019/03/16 11:02:31 AM	System	Log
Door Switches / LowerSlider		        Status	On	Sat 2019/03/16 11:04:04 AM	System	Log
Door Switches / LowerSlider		        Status	Off	Sat 2019/03/16 11:04:09 AM	System	Log
Christmas-Back#				            Status	Off	Sat 2019/03/16 11:07:10 AM	System	Log
Door Switches / LowerSlider		        Status	On	Sat 2019/03/16 11:07:44 AM	System	Log
Door Switches / Costco Rm NEW - Door	Status	Off	Sat 2019/03/16 11:08:08 AM	System	Log
Costco Rm#				                Status	Off	Sat 2019/03/16 11:08:08 AM	System	Log
Door Switches / LowerSlider		        Status	Off	Sat 2019/03/16 11:09:32 AM	System	Log
Door Switches / LowerSlider		        Status	On	Sat 2019/03/16 11:15:50 AM	System	Log
0	null	 					                    Sat 2019/03/16 11:21:23 AM	System	Start

All normal items that happen routinely.   Christmas-back# is not out of place because I used a holiday light plug for a yard extension cord.  Turning it on and in this case off at the device.

[Techman]

The ISY log will only display change of state for your devices. 

Something was using up your ISY resources, not sure exactly what. Possibly your variables?

 

 

[Michel Kohanim]

@MrBill,

Please send your error log to support@universal-devices.com.

With kind regards,
Michel

[MrBill]

On 3/22/2019 at 10:53 AM, Michel Kohanim said:

@MrBill,

Please send your error log to support@universal-devices.com.

With kind regards,
Michel

Sent.

[Techman]

When you get your issue resolved could you please post the results.

[MrBill]

On 3/23/2019 at 5:32 PM, Techman said:

When you get your issue resolved could you please post the results.

Michel identified 3 things from the log.  1)  DNS errors  2) that my wireless tags kumo app was hammering the ISY with updates.  3) that I should update to 5.0.14

After studying the log myself and noticing what he was referring to, I fixed #2 relatively quickly.   I never realized how many updates were arriving.  I had always watched those updates via Event Viewer.  Event viewer apparently only reports the update when the value of the variable actually changes, not when the Tag Manager sends an un-needed update were the value is still current--- in some minutes there could be 30 some API calls spaced seconds apart.  The ISY has in fact been taking this abuse on resources since Nov 2017 (I haven't made changes to the KUMO app since it was originally created)  and just keeps running-- a testament to how well it just keeps running.

I still don't understand what the DNS error looks like.  The associated caution was don't forward UDP ports to the ISY, and use a DNS reservation if I care what the IP is.  Both of those items were negatives to begin with.  I haven't had any ports forwarded at all since I purchased Portal a long time ago (over a year).  Also, I've always used DNS reservations on my network and never given any device a static IP.   As I was writing this post Michel replied one more time, and better explained the DNS situation which was timeouts, after reading that I think it was probably all related to the hammering the ISY was talking from the kumo app and the lack of available resources. 

I also updated to 5.0.14 even tho I don't like to do things like that right before leaving town.  That was a simple process that went without a hitch. I'll also watched that section of the forum so when another release comes out I'll get notified.  I open the admin console probably at least once a week for varying reasons but I am bad about checking/following the forum because I just don't have enough hours in the day.

Thanks for all your help!!

[larryllix]

10 minutes ago, MrBill said:

Michel identified 3 things from the log.  1)  DNS errors  2) that my wireless tags kumo app was hammering the ISY with updates.  3) that I should update to 5.0.14

After studying the log myself and noticing what he was referring to, I fixed #2 relatively quickly.   I never realized how many updates were arriving.  I had always watched those updates via Event Viewer.  Event viewer apparently only reports the update when the value of the variable actually changes, not when the Tag Manager sends an un-needed update were the value is still current--- in some minutes there could be 30 some API calls spaced seconds apart.  The ISY has in fact been taking this abuse on resources since Nov 2017 (I haven't made changes to the KUMO app since it was originally created)  and just keeps running-- a testament to how well it just keeps running.

I still don't understand what the DNS error looks like.  The associated caution was don't forward UDP ports to the ISY, and use a DNS reservation if I care what the IP is.  Both of those items were negatives to begin with.  I haven't had any ports forwarded at all since I purchased Portal a long time ago (over a year).  Also, I've always used DNS reservations on my network and never given any device a static IP.   As I was writing this post Michel replied one more time, and better explained the DNS situation which was timeouts, after reading that I think it was probably all related to the hammering the ISY was talking from the kumo app and the lack of available resources. 

I also updated to 5.0.14 even tho I don't like to do things like that right before leaving town.  That was a simple process that went without a hitch. I'll also watched that section of the forum so when another release comes out I'll get notified.  I open the admin console probably at least once a week for varying reasons but I am bad about checking/following the forum because I just don't have enough hours in the day.

Thanks for all your help!!

How did you resolve the kumoapps hyperactivity?

What does your kumapps code look like?

[MrBill]

4 minutes ago, larryllix said:

How did you resolve the kumoapps hyperactivity?

What does your kumapps code look like?

It was a version of a script I took from one of your posts way back....  I changed it some tho... anyway back when i set that up....

• 7 tags had 10 minute updates
• 3 tags had 1 minute updates

and I was sending Temp, humidity, and battery voltage to the ISY....  so 30 variables total were getting updated.   Tattling on myself, I never got around to using most of that data in ISY programs.  Humidity I didn't think I needed anyway, but i intended to get around to temp alarms and low battery alarms.  Reality strikes tho, and the only two values I was actually using in ISY programs were 2 temperatures to cycle fireplaces.  I just commented out and reconfigured the script/kumo app down to where the only things it updated are the two temperatures that I actually make use of in the ISY. 

I do use the 7 tags with 10 minute updates for refrigerator and freezer temps, except since i never got around to ISY programs I just rely on the Wireless tags app to notify me when temps are out of range... likewise for low battery.

[larryllix]

8 hours ago, MrBill said:

It was a version of a script I took from one of your posts way back....  I changed it some tho... anyway back when i set that up....

• 7 tags had 10 minute updates
• 3 tags had 1 minute updates

and I was sending Temp, humidity, and battery voltage to the ISY....  so 30 variables total were getting updated.   Tattling on myself, I never got around to using most of that data in ISY programs.  Humidity I didn't think I needed anyway, but i intended to get around to temp alarms and low battery alarms.  Reality strikes tho, and the only two values I was actually using in ISY programs were 2 temperatures to cycle fireplaces.  I just commented out and reconfigured the script/kumo app down to where the only things it updated are the two temperatures that I actually make use of in the ISY. 

I do use the 7 tags with 10 minute updates for refrigerator and freezer temps, except since i never got around to ISY programs I just rely on the Wireless tags app to notify me when temps are out of range... likewise for low battery.

LOL! You must have gone through dozens of batteries per year with those 1 minute updates.

I leave mine all at 15 minute updates but you can set specific characteristics of each Tag to update when triggered by that action.
eg: The tag on my garage door updates every 15 minutes but notifies me via KumoApps ----> REST ----> ISY in seconds when the door moves open or closed. The same with leaving my fridge door open too long.  You may need to re-examine your approach there by using the monitoring options in Kumoapps, to avoid killing batteries and swamping your ISY.

The KumoApps line
      tag.updated = function() {
only runs the code to update ISY when something changes so you should only have had updates in ISY when something changed.

I have since installed a heartbeat in that code so I can tell the Tags are even still alive after many minutes of nothing updating. I use five variables for each of my 8 tags without making my ISY busy.

Glad you got your ISY problem isolated now
 

[MrBill]

6 minutes ago, larryllix said:

LOL! You must have gone through dozens of batteries per year with those 1 minute updates.

I leave mine all at 15 minute updates but you can set specific characteristics of each Tag to update when triggered by that action.
eg: The tag on my garage door updates every 15 minutes but notifies me via KumoApps ----> REST ----> ISY in seconds when the door moves open or closed. The same with leaving my fridge door open too long.  You may need to re-examine your approach there by using the monitoring options in Kumoapps, to avoid killing batteries and swamping your ISY.

The KumoApps line
      tag.updated = function() {
only runs the code to update ISY when something changes so you should only have had updates in ISY when something changed.

I have since installed a heartbeat in that code so I can tell the Tags are even still alive after many minutes of nothing updating. I use five variables for each of my 8 tags without making my ISY busy.

Glad you got your ISY problem isolated now
 

In reality the 2 tags with one minute updates for fireplace cycling don't use anymore batteries than the tags with 10 minute updates.  I originally planned to changed the update interval on those two tags seasonally, because 6 months out of the year they provide no function.  Reality I didn't bother.  What does kill batteries faster are the freezers, but i expected that.  Overall battery replacement hasn't bothered me too much.

I don't "arm" any of the tags.  They just push data on the selected update internal. 

 

[larryllix]

I find the batteries last the same length of time in the cold or warm. What does change is the battery voltage and it ma appear to be a low voltage. The tag battery percentage left compensates for the low voltage when they are exposed to cold temperatures. If you have one with a lower voltage you may notice the voltage will come back up when the battery is warmed up again.

"arming" is the term they apply to the Tag for motion sensitivity. Temperature, humidity and lux can be set to a range limit and can update automatically if it falls outside of your set limits. All notifications can be disabled and still trigger the kumoapp updates. I haven't found any good explanations from CAO on this, but have done some experimenting to discover these tags don't have to be set to fast updating, using up the battery life and  "hammering" ISY with constant data.

[palayman]

16 minutes ago, larryllix said:

find the batteries last the same length of time in the cold or warm.

Strange, I get about 10 months out of those inside and 8 months out of the ones outside or in the fridge.