Programs randomly running at the same time each hour throughout the day?

[arzoo]

I have a very strange problem that recently started happening; many of the programs on my eisy have started triggering at the same time, always 52 minutes past the hour, but not every hour. Most of the programs affected seem to be related to my Insteon Keypads and eisy IR controller. I don't see how this could be a hardware failure (eisy or insteon or sensors) because it's always 52 minutes past the hour but on random hours. I'm totally stumped. Any thoughts or suggestions?

[Techman]

You most likely have a program that is either being triggered by a device or another program. Take a look at your log to see if there's any activity at the time your programs are being triggered.  The log will only show if a device turned on, not program activity.

You may have to go through each of your programs to track down the issue.

[arzoo]

The thing is, all of the programs that are triggering have different "If" conditions. There's literally no one device that could cause this behavior.

[arzoo]

I've also looked over the log and it seems like all the buttons on all the Insteon keypads (4) are turning on and then off instantly.

[DennisC]

8 hours ago, arzoo said:

I've also looked over the log and it seems like all the buttons on all the Insteon keypads (4) are turning on and then off instantly.

Are there any white flags next to them in the admin console?

Try waking the keypads up and restoring devices.

[Nestor]

A simple test would be to switch off the keypads and see if the triggering stops. 

[arzoo]

Here's the event viewer from last night at 4:52am. I'm not sure what the RR/OL/ST codes mean but looks like most every device triggered?

[IndyMike]

@arzoo,

• RR = Ramp rate (28 = .5 sec)
• OL = On level (255 = max)
• ST = current status (0=off)

For some reason the status of every device is being refreshed.  I have never seen this. 

A few notes:

1. What you see here is NOT insteon communication.  It is the ISY book keeping current setting for devices.
2. Your event viewer is set to level 1.  It will not catch a command/communication that triggered this event.  Try level 3.
3. Rather than taking a screen shot, try clicking "copy to clipboard" to save the entire file.

Not sure what's causing this, but if you can post a complete event viewer on level 3 we might have a better shot at figuring it out.

 

[Techman]

What's also strange is that the all events are all happening at the exact same time and affecting different devices..  It could be a device that's failing.  You could try disabling the a few of the devices at at time to see if you could narrow it down.

Do you have any programs that run around 4:52 am?

Were all the devices purchased as new?

 

[arzoo]

6 hours ago, IndyMike said:

@arzoo,

• RR = Ramp rate (28 = .5 sec)
• OL = On level (255 = max)
• ST = current status (0=off)

For some reason the status of every device is being refreshed.  I have never seen this. 

A few notes:

1. What you see here is NOT insteon communication.  It is the ISY book keeping current setting for devices.
2. Your event viewer is set to level 1.  It will not catch a command/communication that triggered this event.  Try level 3.
3. Rather than taking a screen shot, try clicking "copy to clipboard" to save the entire file.

Not sure what's causing this, but if you can post a complete event viewer on level 3 we might have a better shot at figuring it out.

 

 

4 hours ago, Techman said:

What's also strange is that the all events are all happening at the exact same time and affecting different devices..  It could be a device that's failing.  You could try disabling the a few of the devices at at time to see if you could narrow it down.

Do you have any programs that run around 4:52 am?

Were all the devices purchased as new?

 

I've attached a level 3 log that I ran a few days earlier. The craziness started around 12:51pm but the cause may have been prior?

No I don't have any programs that run at x:52 on the hour, but I do have programs that remain running with a 1 hour wait state (I don't think they are the cause).

My home automation network has over 100 devices, mostly Insteon switches (dimmer and keypad) and sensors (motion, wet, smoke, siren, etc). I've also got a few Z-wave devices and IR control. Some of the devices are pretty old (~15 years) and others much newer. I've replaced a lot of the Insteon stuff as it fails (including the PLM).

As for programs I've got close to 200 and I've done some pretty extensive stuff (imo).

After looking over the level 3 log I decided to start by disabling a motion sensor that seemed to trigger for no reason. But I still don't see how any one device could cause this behavior?

Anyway, thanks for everyone's help with this!

ISY-Events-Log.v5.7.1__Fri 2024.07.26 01.15.58 PM.txt

[Techman]

Take a look at this article

https://wiki.universal-devices.com/INSTEON_Random_All_On_Events

Insteon Motion sensors with low batteries can wreak havoc on an Insteon network, especially the 1st generation sensor.  What is the (6 digit) address of your motion sensors? Do they show any activity around the time your issue starts

It could also be one of your older single band Insteon devices that's failing, especially if it's 15 years old. The newer devices have improved firmware and hardware and are less prone to failing.

 

[arzoo]

46 minutes ago, Techman said:

Take a look at this article

https://wiki.universal-devices.com/INSTEON_Random_All_On_Events

Insteon Motion sensors with low batteries can wreak havoc on an Insteon network, especially the 1st generation sensor.  What is the (6 digit) address of your motion sensors? Do they show any activity around the time your issue starts

It could also be one of your older single band Insteon devices that's failing, especially if it's 15 years old. The newer devices have improved firmware and hardware and are less prone to failing.

 

I do have a few older Insteon motion sensors, including the one I currently disabled. So far no crazy stuff but it typically happens overnight (?). Thanks again for the help.

[IndyMike]

@arzoo,  your log took awhile to decipher.  There's a lot going on.  I've tried to categorize things below to make them a little more understandable.

Section 1 shows the ISY transmitting an ON command to group 25 IMMEDIATELY followed by an OFF command.  This was likely a program executing because the commands were processed so quickly.  Normal processing would show the ISY command [INST-TS-I1] immediately followed by the PLM ACK [INST-ACK].  

This may have been a programming typo where you intended to execute a scene OFF command 2x.  If not, you may have 2 programs that are triggering each other.

Section 2 shows the ISY bookkeeping the On/Off states for the devices in Group 25.  By my count, there are 65 unique devices in that group.  I find it odd that the ISY is statusing the Ramp Rate and On Level, but that could be a new feature of the EISY (I have the ISY994).  I also find it odd that all of the KPL sub buttons appear to be in Group 25.  As I said, there's a lot going on.

Keep in mind that NO commands are issued in section 2.  There's 600 lines of log entry that occurs in 0 elapsed time with no commands issued or received.

Section 3 I flat don't understand.  Many OFF commands to different groups multiple times.  NONE of these are acknowledged by the PLM.  I don't understand why they are not acknowledged - they appear to be valid commands.  I also don't understand why you would turn that many scenes off in rapid succession, mutiple times.  Again, because they happened to quickly, these are probably coming from a program.

Very ODD to say the least.

Section 4 shows a device direct command requesting that the device BEEP.  It's very normal with both a PLM and Device Acknowledge.

 

Conclusion:  This does not appear to be a classic All-On event.  A all on event is purportedly caused by a command collision at the PLM.  Because the event is the result of a collision,  the ISY DOES NOT HAVE KNOWLEDGE of the event.  Your logs show that the EISY does have knowledge of the event and is trying to reflect that status of the devices.

Since the event appears program related I would start by inspecting/disabling programs that work with Group 25 (On/OFF in Section 1). 

You may also want to inspect the program(s) that are issuing the multiple group OFF commands in Section 3.  I don't understand why, but the PLM is not responding to these commands.

Section 1 - Group 25 turned on then immediately turned off   Fri 07/26/2024 12:51:42 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 11 00 Group 25 On Fri 07/26/2024 12:51:42 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off Fri 07/26/2024 12:51:42 PM : [INST-ACK    ] 02 62 00.00.25 CF 11 00 06          LTONRR (00) PLM Ack Fri 07/26/2024 12:51:42 PM : [INST-ACK    ] 02 62 00.00.25 CF 13 00 06          LTOFFRR(00) PLM ACK     Section 2 - ISY bookkeeps the on/off status of 65 unique devices + secondary buttons   Fri 07/26/2024 12:51:42 PM : [55 FF B6 6      ]       ST 255 (uom=100 prec=0) ISY Book keeping ON/OFF  Fri 07/26/2024 12:51:42 PM : [55 FF B6 6      ]       ST   0 (uom=100 prec=0) 65 Unique devices Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       RR  28 (uom=25 prec=0)   Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       OL 255 (uom=100 prec=0)   Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       ST 255 (uom=100 prec=0)       Section 3 - Multiple scenes turned off - PLM doesn not respond to these commands   Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 3A CF 13 00 Group 3A Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 42 CF 13 00 Group 42 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 43 CF 13 00 Group 43 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 41 CF 13 00 Group 41 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 41 CF 13 00 Group 41 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 42 CF 13 00 Group 42 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 43 CF 13 00 Group 43 Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off  Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 4B CF 13 00 Group 4B Off Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off      Section 4 ? Odd beep command to device 55.FF.B6 both PLM and Device Ack   Fri 07/26/2024 12:51:59 PM : [INST-ACK    ] 02 62 55.FF.B6 0F 30 00 06          BEEP   (00)   Fri 07/26/2024 12:51:59 PM : [INST-SRX    ] 02 50 55.FF.B6 43.6E.51 2F 30 00    BEEP   (00)   Fri 07/26/2024 12:51:59 PM : [Std-Direct Ack] 55.FF.B6-->ISY/PLM Group=0, Max Hops=3, Hops Left=3   Group 25 Devices - Does not include sub-buttons

1 14 72 B6 2 14 7B DF 3 15 C7 F5 4 16 9E 2F 5 16 9E E  6 16 A0 74 7 16 A2 45 8 16 C5 51 9 16 C7 9B 10 16 CA DA 11 16 CC 2A 12 16 CC 31 13 16 CC E  14 16 CD 0  15 16 CD D9 16 18 0 1E  17 18 9F 2E 18 19 CE BC 19 1A 1E BD 20 1A 21 CA 21 1A 27 8A 22 1A 53 6A 23 1A 5F DC 24 1A AC 83 25 1A AF 70 26 1F 73 67 27 20 E7 6D 28 20 EA 46 29 24 6B C1 30 24 96 D7 31 25 4E C2 32 25 5B FA 33 25 B8 99 34 27 6E 30 35 27 70 AE 36 27 70 D6 37 29 40 87 38 2B 58 31 39 2B 5C F0 40 2B 65 B4 41 2B 6E 7E 42 3A FB D4 43 3D F4 FF 44 3D F9 1B 45 3D FE D6 46 3E 10 47 47 3E 8 E1  48 3E E EF  49 4B DE 4E 50 4C 85 3D 51 4E 27 F9 52 4E 29 6F 53 4F A0 20 54 50 AD 58 55 50 B0 CA 56 50 B9 76 57 51 74 E5 58 52 E0 A6 59 53 28 E5 60 53 29 8B 61 54 55 53 62 54 57 A3 63 54 E3 2F 64 55 FF B6 65 70 7B 5B

[arzoo]

@IndyMike Thanks SO much for helping decipher the log! I'm on vacation for the week so I'll get back to this when I get home. I will say this, since replacing the batteries in the one suspect motion sensor, the problem has not reoccurred.

[arzoo]

Apparently the motion sensor was not the issue, 1:52 a.m. everything triggered again last night. Back to the drawing board. Just so odd that it's always the same time every night. I'll have to review all my programs and see if I have anything scheduled to run at that time, but I'm pretty sure I don't.

[IndyMike]

32 minutes ago, arzoo said:

Apparently the motion sensor was not the issue, 1:52 a.m. everything triggered again last night. Back to the drawing board. Just so odd that it's always the same time every night. I'll have to review all my programs and see if I have anything scheduled to run at that time, but I'm pretty sure I don't.

That's unfortunate.  Do you have any programs that re-trigger on an hourly basis?  The start times can drift if delayed by other events or if they are qualified by sunrise/sunset.  I've seen programs "drift together" and cause issues.

I have a number of programs that Poll devices throughout the day -

Flood Night Poll - [ID 006E][Parent 0002][Run At Startup]

If
        From    Sunset  +  1 minute
        To      Sunrise - 10 minutes (next day)
    And Time is Last Run Time for 'Flood Night Poll' + 15 minutes
 
Then
        Send X10 'F1/Status Request (10)'
        Wait  9 seconds
        Send X10 'F3/Status Request (10)'
        Wait  9 seconds
        Send X10 'F5/Status Request (10)'
 
Else
   - No Actions - (To add one, press 'Action')
 

 

[Geddy]

@arzoo you could use the raw text portion of the find/replace option in the programs to potentially find a time reference. Or I would suggest you copy the root folder to the clipboard then paste in a text editor and search for time or other triggers in the text of all the programs. (right click on the root folder and select COPY folder to clipboard - NOT "export")

Have you checked the summary tab for the programs? Anything showing "last run" at the time this happens? You originally said it was happening at random hours so maybe you'll see it during the day rather than 1am time. 

 

[arzoo]

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

[paulbates]

28 minutes ago, arzoo said:

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

There's a lot of symptoms and great analysis in this post... but if it is a flaky Insteon MS, I've found that letting the battery run down can confuse the MS (at least the MS1) and cause unpredictable behavior. I made it a practice when I had them to factory reset and "restore device" when replacing a low/depleted battery. I got many years with them when staying ahead of the battery replacements, in fact sold them with the house.

[IndyMike]

4 hours ago, arzoo said:

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

Can you provide the Insteon address for the motion sensor?  I can check your event log to see if it sent something odd.

I agree with @paulbates regarding the factory reset of the MS1's.  This has become standard practice for me whenever I replace a battery.

[arzoo]

On 8/2/2024 at 2:25 PM, IndyMike said:

Can you provide the Insteon address for the motion sensor?  I can check your event log to see if it sent something odd.

14 72 B6 1

But the issue continues even with the sensor disabled. I'll have to try some of the other suggestions to continue debugging. I'll focus on the fact that it only happens at 52 minutes past the hour.

[Techman]

Your motion sensor (14 72 B6) appears in almost every log entry, so chances are this is the device that's causing you problems.  Disabling the device does not stop it from sending spurious data into the Insteon network. You need to remove its battery in order to completely disable it.

You should also look for a program that may be running around the time of your issues

Edited August 3 by Techman

[arzoo]

3 hours ago, Techman said:

Your motion sensor (14 72 B6) appears in almost every log entry, so chances are this is the device that's causing you problems.  Disabling the device does not stop it from sending spurious data into the Insteon network. You need to remove its battery in order to completely disable it.

You should also look for a program that may be running around the time of your issues

That's good to hear and will do when we get home! Thanks

[IndyMike]

On 8/3/2024 at 3:05 PM, arzoo said:

14 72 B6 1

But the issue continues even with the sensor disabled. I'll have to try some of the other suggestions to continue debugging. I'll focus on the fact that it only happens at 52 minutes past the hour.

Thanks - that helps.

As @Techman indicated, the 14.72.B6 sensor shows up a lot.  It appears to have an on/off time of 30 seconds.  It is being evaluated by  8 separate programs.  One (or two) of those programs is turning on scene 25 and then immediately turning it off again.  Some of these may be disabled - they will still show up in the log.

There appears to be a second sensor (14.7B.DF) that is also activating.  This one appears to have an on/off interval of 1 minute.  It is also being evaluated in 8 separate programs with two shared programs (highlighted green below).  Again, there is a program turning on scene 25 and the immediately turning it off again.

There is a 3rd device that is turning on/off multiple scenes.  I can't tell the device type.  The device is evaluated by 6 different programs. 

All three of the above devices were active at the 12:51:59 mark when things went out the window.  You should be able to search for programs where these devices are used by using the "find" feature as @Geddy described.  I have a feeling you have multiple programs firing at the same time and causing issues.

Motion sensors are notorious for firing multiple communications per event.  Both of your devices send 2 On messages + 1 cleanup per event.  As a precaution, you may want to add a wait to your programs to prevent collisions. 

I still do not see a smoking gun anywhere that would indicate what triggered the "event".  Try some precautionary changes and things may become clearer.  This could also be caused by a non-insteon event (PG3, rest interface, Home Assistant) that would not show up in the Event Viewer.

 

 

 

 

 

 

[arzoo]

I removed the battery from the suspected sensor (14 72 B6 1) but the problem triggered again last night at 12:52. The fact that it's always around the same time has me thinking what programming I might have in place that's the culprit. The only programs I'm running with an hourly trigger are for monitoring the heartbeat signal on four Insteon water sensors. When the water sensors are working correctly, they send a heartbeat once about every 24 hours. I have programs that check every hour, incrementing a counter. If the counter exceeds 24 then I assume the battery is low. Anyway, I'll disable those programs and see if it makes any difference.