Programs randomly running at the same time each hour throughout the day?

[Techman]

@arzoo

Make sure all your variables are STATE and not INTEGER

Also verify that all your variables in the programs are correct, and their state is correctly shown in the variable setup

[IndyMike]

20 hours ago, arzoo said:

I'm unsure what's causing the rest activity; maybe the UD Mobile app on my phone?

I agree, I may not have disabled all programs for long enough but the problem has been happening pretty consistently at least one time every morning at 12:52 or 1:52 or 2:52am, so an overnight test seems valid.

I re-enabled all but two folder programs (one for alarm stuff and one for alexa stuff) and the problem happened again, so I disabled the alarm folder (leaving only alexa folder enabled) and the problem did not occur over night. I realize these are very short test periods.

What I've done now is enable all programs folders except for the alarm programs. I'll continue to run in this configuration and see how it goes. I've already looked over the programs in that folder and I don't see how any of them could cause the x:52 activity, so that's still a mystery.

You asked what happens when the problem occurs - the crazy stuff; I have a set of programs that function as an addition to my actual house alarm system. Were the main house alarm to trigger, these programs do stuff like flash all the lights (inside and outside), turn on additional Insteon sirens, beep at all the keypads, and send text messages. The text messages were the first indication that something was not right - I started getting a text message (at 52 minutes past) that the alarm mode was disabled and the Insteon keypad in our bedroom would beep (waking me up). As you've mentioned, the log also indicates a bunch of other Insteon scenes and switches are turning on and off, but I've not actually seen any physical indication of this. There's also indication that IR commands are firing (from our home theater remote I send IR commands to the EISY to activate lighting scenes), but here again I've not seen any physical indication of this. And based on what you've said prior, it seems like there are no triggers for all these events when the problem occurs?

Sorry, I missed this yesterday...

An alarm program may make sense if it is triggered by a non-insteon device.  The Event viewer would not "see" the trigger.

The event log shows many scenes being turned off, and 1 device being Beeped  (55.FF.B6).  Since this is a device direct command with a specific address, you should be able to use the "Find/Replace" function to locate programs that call this device.

Wed 08/07/2024 02:52:11 AM : [INST-ACK    ] 02 62 55.FF.B6 0F 30 00 06          BEEP   (00)

 

[arzoo]

I'm ready to throw in the towel here. I've gone through all my programs and there's just no way they could be the cause of stuff going crazy only at 52 past some random hour. Same for the devices, even the motion sensors - no way they could only fail or trigger or whatever only at 52 past some random hour. One last hail mary would be to replace the PLM.

[IndyMike]

Not my place to disagree, but I thought the process of disabling programs was showing promise.

I don't know of any PLM function that would automatically activate programs at 52 past the hour.  The PLM is just a hardware modem.  It has no timer capability. 

If you proceed with the PLM replacement I would advise 3 things in addition to the normal procedures (backup, etc):

1. Make sure you have good communications going in (Hops remaining >= 2 when querying devices).
2. Pick a low activity time so people aren't tripping motion sensors and activating lights (early AM is good around my house).
3. Use your "program disable" to make sure that the 52 after event doesn't activate during the replacement.

[arzoo]

45 minutes ago, IndyMike said:

Not my place to disagree, but I thought the process of disabling programs was showing promise.

I don't know of any PLM function that would automatically activate programs at 52 past the hour.  The PLM is just a hardware modem.  It has no timer capability. 

If you proceed with the PLM replacement I would advise 3 things in addition to the normal procedures (backup, etc):

1. Make sure you have good communications going in (Hops remaining >= 2 when querying devices).
2. Pick a low activity time so people aren't tripping motion sensors and activating lights (early AM is good around my house).
3. Use your "program disable" to make sure that the 52 after event doesn't activate during the replacement.

No, please disagree, you've been very helpful. I guess I just feel over my head and was venting a bit lol.

I thought I was making progress by disabling the folders but then the results were not consistent. For example, with the top level folder disabled, it seemed like the problem did not occur - no crazy activity at 52 past. Then I re-enabled the top level and disabled all sub folders except two and the problem came back. So you would think the program(s) in one of those two folders would be the cause. But after running tests with only one of those two folders enabled at a time (and all others disabled), in both cases the problem seemed to occur which puts me back to square one.

And I still just don't see how any program could cause the problem at such a specific and yet random time? Which makes me think could it be the eisy somehow? I unplugged the eisy and the plm for about 30 minutes and then powered them back up. I figure if the eisy is the reason the problem only happens at 52 past, then powering down and back up at a different time should shift the time when the problem occurs. I'll report back when it does.

[Javi]

I did not read through the entire thread but I've seen similar issues through the years.  While most were related to Programs, there are a few less common things to consider.

• Do you have a query program?  It's possible eisy is gaining or losing (a) device's state. A query then triggers a program when query is run or when device finally reports status after query cleared the device's status from Unknown State.
• A program is in a vicious loop using all available resources.  This causes may strange issues.  I've seen this happen where issues start at the same time everyday, then there is something that either automatically or manually stops the loop sometime later.   The user cannot replicate issue again until the loop starts.  This usually correlates to UD Mobile being very slow.
• User at one point had IFTTT/Alexa/Tasker/etc issue a command to a program/variable and this is still happening.  The program/variable may have been deleted, however ISY will reuse program/variable ID's if they are deleted, so the old automation appeared to be removed until the user added a new program/variable which reused the old ID and the automation now triggers the new program/variable.  If you find a program which may be the culprit, try copy then delete the original so the program has a new ID.
• eisy is rebooting. Usually due to a program in a vicious loop or too many requests, similar to denial of service. Too many requests were usually caused by a malfunctioning integration like home assistant or a PG3 node server which the user has allowed ISY access.  This usually correlates to UD Mobile not being able to connect.
• Geofencing is triggering a program/variable with a false entry/exit.

[IndyMike]

39 minutes ago, arzoo said:

No, please disagree, you've been very helpful. I guess I just feel over my head and was venting a bit lol.

I thought I was making progress by disabling the folders but then the results were not consistent. For example, with the top level folder disabled, it seemed like the problem did not occur - no crazy activity at 52 past. Then I re-enabled the top level and disabled all sub folders except two and the problem came back. So you would think the program(s) in one of those two folders would be the cause. But after running tests with only one of those two folders enabled at a time (and all others disabled), in both cases the problem seemed to occur which puts me back to square one.

And I still just don't see how any program could cause the problem at such a specific and yet random time? Which makes me think could it be the eisy somehow? I unplugged the eisy and the plm for about 30 minutes and then powered them back up. I figure if the eisy is the reason the problem only happens at 52 past, then powering down and back up at a different time should shift the time when the problem occurs. I'll report back when it does.

I understand venting.  I do it often - it's a healthy release...

I understand your thought process regarding the programs.  The difficult part is we don't understand the trigger, so we can't really asses whether the problem has gone away without significant time.  There may also be more than 1 program that is being activated (more than 1 folder affected).

I am not well versed on plug-ins, mobile-linc and the like.  I don't know if these could be the trigger for the 52 after events.

What type of alarm are you using?  What type of input devices (hardwired or rf)?

Your "powering down" to time shift polling programs is a good thought.  Please do report back on the results.

Since the "beep" seems common to the occurrence, you could search programs that perform this function and work backward.

Just saw @Javi's response.  Please pay particular attention - these are next level "failure mode" types of upsets that most of us never consider.

Lastly - I would consider opening a ticket before you replace the PLM.  My comment about replacing the PLM was based on the fact that it was loosing it's link tables.  That doesn't appear to have happened to your recently and I would discount this as the source of your current issues. At this point I feel that replacing the PLM would only further confuse things. 

Edited August 10 by IndyMike

[arzoo]

I powered down the EISY and PLM (for 30 minutes) a few days ago and thought somehow that might have resolved the issue as nothing went crazy for two nights in a row. But then last night at 1:52 so back to the drawing board.

Edited August 12 by arzoo

[arzoo]

I'm confused as to why the EISY would not include info in the event viewer to indicate that a program executed? This seems like a simple feature that would be very helpful for debugging? Why just add empty timestamps?

[arzoo]

Looking over the log from last night when the problem occurred and I see this entry:

Mon 08/12/2024 01:51:57 AM : [D2D-CMP 004B] STS [55 FF B6 8] ST Converted values Event=0 Condition=0 prec=0
Mon 08/12/2024 01:51:57 AM : [D2D-CMP 004B] STS [55 FF B6 8] ST op=1 Event(val=0 uom=100 prec=0) is Condition(val=0 uom=51 prec=0) --> true

I'm assuming this indicates that program ID 004B was triggered and ran the ELSE clause (IF evaluated as true), yes? This program is in a disabled folder and the program itself is disabled and no other program calls this program, so how is this possible?

Edited August 12 by arzoo

[paulbates]

7 minutes ago, arzoo said:

This program is in a disabled folder and the program itself is disabled and no other program calls this program, so how is this possible?

Another root cause to unpredictable, flaky behavior in an older ISY is the SIM card going bad. This potentially is a cause.

I didn't scroll back through this thread to see if this was covered or not... but *it is* something that will eventually happen. If you search the forum for SIM Card, or the wiki the procedure is covered. 

The problems you are having still seem systemic to me based on their random nature.. ed corrupt PLM or/and corrupt SIM, and not a program suddenly going bad

Edited August 12 by paulbates

[IndyMike]

9 minutes ago, arzoo said:

Looking over the log from last night when the problem occurred and I see this entry:

Mon 08/12/2024 01:51:57 AM : [D2D-CMP 004B] STS [55 FF B6 8] ST Converted values Event=0 Condition=0 prec=0
Mon 08/12/2024 01:51:57 AM : [D2D-CMP 004B] STS [55 FF B6 8] ST op=1 Event(val=0 uom=100 prec=0) is Condition(val=0 uom=51 prec=0) --> true

I'm assuming this indicates that program ID 004B was triggered and ran the ELSE clause (IF evaluated as true), yes? This program is in a disabled folder and the program itself is disabled and no other program calls this program, so how is this possible?

The program comparison is being performed (IF statement check on 55.FF.B6).  If the program or folder is disabled, the THEN/ELSE will NOT be run.

This is not an error.  My 994I does the same.

Edited August 12 by IndyMike

[IndyMike]

2 hours ago, arzoo said:

I'm confused as to why the EISY would not include info in the event viewer to indicate that a program executed? This seems like a simple feature that would be very helpful for debugging? Why just add empty timestamps?

@arzoo, you appear to have a program firing every 30 seconds and generating the timestamps.  That's a pretty tight loop.  You should be able to locate the program by looking at the program summary tab.

You also have a REST command executing every 5 minutes.  I'm not sure if this is program related or an external communication. 

[arzoo]

32 minutes ago, paulbates said:

Another root cause to unpredictable, flaky behavior in an older ISY is the SIM card going bad. This potentially is a cause.

I'm using a relatively new EISY; would the SIM card still be an issue?

[arzoo]

40 minutes ago, IndyMike said:

The program comparison is being performed (IF statement check on 55.FF.B6).  If the program or folder is disabled, the THEN/ELSE will NOT be run.

This is not an error.  My 994I does the same.

Cool, I won't worry about this then. Thanks!

27 minutes ago, IndyMike said:

@arzoo, you appear to have a program firing every 30 seconds and generating the timestamps.  That's a pretty tight loop.  You should be able to locate the program by looking at the program summary tab.

You also have a REST command executing every 5 minutes.  I'm not sure if this is program related or an external communication. 

Hmm, I'll have to try and figure out which program is running in a 30 second loop but I'm pretty sure I've never configured any like that. As for the REST command, that's another mystery; maybe the UD Mobile app on my phone?

Update: I'm sitting here watching the event viewer indicating a program is running every 30 seconds, yet the program summary tab shows the last program to run was 15 minutes ago?

Edited August 12 by arzoo

[Geddy]

19 minutes ago, arzoo said:

I'm using a relatively new EISY; would the SIM card still be an issue?

I think @paulbates meant SD CARD (not “sim”) on the ISY994. There isn’t an SD card on the eisy, it’s an actual hard drive system. It very well could have been an indicator of a failing SD card on the old system. But doubt it’s impacting the eisy, but I do wonder if it could have corrupted something prior to the migration. 

 

40 minutes ago, IndyMike said:

arzoo, you appear to have a program firing every 30 seconds and generating the timestamps.  That's a pretty tight loop.  You should be able to locate the program by looking at the program summary tab.

I think this is “normal” for IoX now. It’s strange because the time is offset from the timestamp. This has been noted various times soon after the eisy was available and appears to impact Polisy and eisy. I haven’t looked at the ISY994 to see if it has started these timestamp events in the log, but have seen it on my eisy and prior Polisy.

@arzoo It’s not a program running. 

[arzoo]

17 minutes ago, Geddy said:

I think this is "normal” for IoX now. It’s strange because the time is offset from the timestamp. This has been noted various times soon after the eisy was available and appears to impact Polisy and eisy. I haven’t looked at the ISY994 to see if it has started these timestamp events in the log, but have seen it on my eisy and prior Polisy.

@arzoo It’s not a program running. 

Good to know, one less mystery. Thanks!

I turned off my phone for a bit and still getting the "Create REST U7" events every 5 minutes, so not the UD Mobile app. Maybe this is also normal for the IoX? Someone else also posted a question about this but there was no answer.

Edited August 12 by arzoo

[Geddy]

@arzoo with all the previous help and troubleshooting you've been doing to still have issue it might be time to open a support ticket with UD to see if they can pinpoint the possible issue. 

https://www.universal-devices.com/my-tickets

Keep troubleshooting stuff and let us know what comes from the support ticket. Hopefully it's an easy fix once working directly with them. 

 

 

1 hour ago, arzoo said:

I turned off my phone for a bit and still getting the "Create REST U7" events every 5 minutes

Are you running any node servers? I just opened up admin console to take a look at the event viewer.  I'm not seeing the time events like I recall seeing in the past (make sure you've updated IoX and PG3x to their current versions). 

I did get a sloth of "Create REST U7" hits in the event viewer, but they all seemed to come from node server connections:

Quote

Mon 08/12/2024 12:06:38 PM : [D2D EVENT       ] Event [n008_setup] [DON] [0] uom=0 prec=-1
Mon 08/12/2024 12:06:38 PM : [n008_setup      ]      DON   0
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/3/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/4/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/5/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/6/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/8/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/10/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/20/connection]
Mon 08/12/2024 12:07:37 PM : Create REST U7 [/rest/profiles/ns/0/connection]
 

I would probably ignore those and not worry about them if they just have this appearance. But I'm not 100% sure of that...just based off my system and a few minutes of having event viewer open on level 3. 

[IndyMike]

@arzoo, I agree with @Geddy that it's probably time to submit a ticket. 

We've not been able to identify an Insteon trigger for your events.  Possible that things are being initiated by a node server/rest event.  Best to get the experts involved.

[arzoo]

1 hour ago, Geddy said:

@arzoo with all the previous help and troubleshooting you've been doing to still have issue it might be time to open a support ticket with UD to see if they can pinpoint the possible issue. 

https://www.universal-devices.com/my-tickets

Keep troubleshooting stuff and let us know what comes from the support ticket. Hopefully it's an easy fix once working directly with them. 

 

 

Are you running any node servers? I just opened up admin console to take a look at the event viewer.  I'm not seeing the time events like I recall seeing in the past (make sure you've updated IoX and PG3x to their current versions). 

I did get a sloth of "Create REST U7" hits in the event viewer, but they all seemed to come from node server connections:

I would probably ignore those and not worry about them if they just have this appearance. But I'm not 100% sure of that...just based off my system and a few minutes of having event viewer open on level 3. 

I don't believe I'm running any node servers. As for my EISY version, I just now upgraded from 5.7.1 to the latest (5.8.4). Not sure why I didn't think to do this sooner. I'll keep an eye on the event log and see if those empty program events ever 30 seconds have gone away (so far I'm not seeing any).

Before I open a ticket I'm trying one last debug attempt; Looking back to my last EISY backup, I had replaced a defective Insteon IOLinc 2450 sensor. The sensor is hard-wired to the siren on my home alarm system - this is how I detect when the alarm is tripped. This got me thinking, could the actual alarm system be "doing something" at 52 past the hour and that (maybe combined with another faulty sensor or device?) is the cause of the crazy shit on the EISY? Also, that sensor is plugged into the same outlet as the PLM. I know it a long shot but at this point I'll try anything. I've disconnected the sensor and we'll see what happens tonight.

Edited August 12 by arzoo

[arzoo]

It's been three days since I unplugged the IOLinc 2450 sensor and the problem has not re-occurred and there's been nothing out of the ordinary in the event viewer. Next up I'll try I'll try plugging the sensor back in, but this time to a different outlet than the PLM. I don't see how the outlet would make any difference but I figure I'll give it a try. Assuming the problem comes back then I'm pretty confident the sensor is defective and I'll order a new one.

[IndyMike]

29 minutes ago, arzoo said:

It's been three days since I unplugged the IOLinc 2450 sensor and the problem has not re-occurred and there's been nothing out of the ordinary in the event viewer. Next up I'll try I'll try plugging the sensor back in, but this time to a different outlet than the PLM. I don't see how the outlet would make any difference but I figure I'll give it a try. Assuming the problem comes back then I'm pretty confident the sensor is defective and I'll order a new one.

I do hope you've found your culprit.  It's difficult for me to describe my dislike for the IOLinc without using language that would get me banned from this forum.  I would very much advise that you find a different sensor to use as the trigger.  Describe the application and there are many members that can help.

As you indicated, I don't see any reason that having the sensor in the same outlet as the PLM would be a problem.  The real problem is using the IOLinc ANYWHERE in your system.  They are simply failure prone devices.

Could you provide the address for the IOLinc - I'd like to check your event viewer posts for activity.

[arzoo]

10 minutes ago, IndyMike said:

I do hope you've found your culprit.  It's difficult for me to describe my dislike for the IOLinc without using language that would get me banned from this forum.  I would very much advise that you find a different sensor to use as the trigger.  Describe the application and there are many members that can help.

As you indicated, I don't see any reason that having the sensor in the same outlet as the PLM would be a problem.  The real problem is using the IOLinc ANYWHERE in your system.  They are simply failure prone devices.

Could you provide the address for the IOLinc - I'd like to check your event viewer posts for activity.

The IOLinc is 70.7B.5B

The original one lasted 15 years until it failed back in May. When I replaced it I went with one listed as new open box (or something like that) to save $ which may have been a mistake. Although from what you're saying I should avoid the IOLinc entirely and go with a different manufacturer - any suggestions?

[IndyMike]

@arzoo, can you describe how you're using the IOLinc sensor input.  You mention that it's "Hardwired" to the alarm siren??  Voltage sensing? Contact closure?

Do you have any Zwave or Zigbee devices in your system?

Edited August 15 by IndyMike

[paulbates]

While I understand where indymike is coming from, there a pluses for it being directly integrated with my Insteon network if the Insteon network is strong. Like you I've used them for long periods at my last house. I have them monitoring my sump pump run time and depth alarm at this house and it works well for that purpose.

To follow the diagnostic path you've been on, I would plug it back in, but disconnect the input wires to the iolinc and retry. If it reappears, its the iolinc... if it doesn't, its potentially the alarm system or wiring.

Edited August 15 by paulbates