Jump to content

Programs randomly running at the same time each hour throughout the day?


Recommended Posts

I have a very strange problem that recently started happening; many of the programs on my eisy have started triggering at the same time, always 52 minutes past the hour, but not every hour. Most of the programs affected seem to be related to my Insteon Keypads and eisy IR controller. I don't see how this could be a hardware failure (eisy or insteon or sensors) because it's always 52 minutes past the hour but on random hours. I'm totally stumped. Any thoughts or suggestions?

Link to comment

You most likely have a program that is either being triggered by a device or another program. Take a look at your log to see if there's any activity at the time your programs are being triggered.  The log will only show if a device turned on, not program activity.

You may have to go through each of your programs to track down the issue.

Link to comment
8 hours ago, arzoo said:

I've also looked over the log and it seems like all the buttons on all the Insteon keypads (4) are turning on and then off instantly.

Are there any white flags next to them in the admin console?

Try waking the keypads up and restoring devices.

Link to comment

@arzoo,

  • RR = Ramp rate (28 = .5 sec)
  • OL = On level (255 = max)
  • ST = current status (0=off)

For some reason the status of every device is being refreshed.  I have never seen this. 

A few notes:

  1. What you see here is NOT insteon communication.  It is the ISY book keeping current setting for devices.
  2. Your event viewer is set to level 1.  It will not catch a command/communication that triggered this event.  Try level 3.
  3. Rather than taking a screen shot, try clicking "copy to clipboard" to save the entire file.

Not sure what's causing this, but if you can post a complete event viewer on level 3 we might have a better shot at figuring it out.

 

Link to comment

What's also strange is that the all events are all happening at the exact same time and affecting different devices..  It could be a device that's failing.  You could try disabling the a few of the devices at at time to see if you could narrow it down.

Do you have any programs that run around 4:52 am?

Were all the devices purchased as new?

 

Link to comment
6 hours ago, IndyMike said:

@arzoo,

  • RR = Ramp rate (28 = .5 sec)
  • OL = On level (255 = max)
  • ST = current status (0=off)

For some reason the status of every device is being refreshed.  I have never seen this. 

A few notes:

  1. What you see here is NOT insteon communication.  It is the ISY book keeping current setting for devices.
  2. Your event viewer is set to level 1.  It will not catch a command/communication that triggered this event.  Try level 3.
  3. Rather than taking a screen shot, try clicking "copy to clipboard" to save the entire file.

Not sure what's causing this, but if you can post a complete event viewer on level 3 we might have a better shot at figuring it out.

 

 

4 hours ago, Techman said:

What's also strange is that the all events are all happening at the exact same time and affecting different devices..  It could be a device that's failing.  You could try disabling the a few of the devices at at time to see if you could narrow it down.

Do you have any programs that run around 4:52 am?

Were all the devices purchased as new?

 

I've attached a level 3 log that I ran a few days earlier. The craziness started around 12:51pm but the cause may have been prior?

No I don't have any programs that run at x:52 on the hour, but I do have programs that remain running with a 1 hour wait state (I don't think they are the cause).

My home automation network has over 100 devices, mostly Insteon switches (dimmer and keypad) and sensors (motion, wet, smoke, siren, etc). I've also got a few Z-wave devices and IR control. Some of the devices are pretty old (~15 years) and others much newer. I've replaced a lot of the Insteon stuff as it fails (including the PLM).

As for programs I've got close to 200 and I've done some pretty extensive stuff (imo).

After looking over the level 3 log I decided to start by disabling a motion sensor that seemed to trigger for no reason. But I still don't see how any one device could cause this behavior?

Anyway, thanks for everyone's help with this!

ISY-Events-Log.v5.7.1__Fri 2024.07.26 01.15.58 PM.txt

Link to comment

Take a look at this article

https://wiki.universal-devices.com/INSTEON_Random_All_On_Events

Insteon Motion sensors with low batteries can wreak havoc on an Insteon network, especially the 1st generation sensor.  What is the (6 digit) address of your motion sensors? Do they show any activity around the time your issue starts

It could also be one of your older single band Insteon devices that's failing, especially if it's 15 years old. The newer devices have improved firmware and hardware and are less prone to failing.

 

Link to comment
46 minutes ago, Techman said:

Take a look at this article

https://wiki.universal-devices.com/INSTEON_Random_All_On_Events

Insteon Motion sensors with low batteries can wreak havoc on an Insteon network, especially the 1st generation sensor.  What is the (6 digit) address of your motion sensors? Do they show any activity around the time your issue starts

It could also be one of your older single band Insteon devices that's failing, especially if it's 15 years old. The newer devices have improved firmware and hardware and are less prone to failing.

 

I do have a few older Insteon motion sensors, including the one I currently disabled. So far no crazy stuff but it typically happens overnight (?). Thanks again for the help.

Link to comment

@arzoo,  your log took awhile to decipher.  There's a lot going on.  I've tried to categorize things below to make them a little more understandable.

Section 1 shows the ISY transmitting an ON command to group 25 IMMEDIATELY followed by an OFF command.  This was likely a program executing because the commands were processed so quickly.  Normal processing would show the ISY command [INST-TS-I1] immediately followed by the PLM ACK [INST-ACK].  

This may have been a programming typo where you intended to execute a scene OFF command 2x.  If not, you may have 2 programs that are triggering each other.

Section 2 shows the ISY bookkeeping the On/Off states for the devices in Group 25.  By my count, there are 65 unique devices in that group.  I find it odd that the ISY is statusing the Ramp Rate and On Level, but that could be a new feature of the EISY (I have the ISY994).  I also find it odd that all of the KPL sub buttons appear to be in Group 25.  As I said, there's a lot going on.

Keep in mind that NO commands are issued in section 2.  There's 600 lines of log entry that occurs in 0 elapsed time with no commands issued or received.

Section 3 I flat don't understand.  Many OFF commands to different groups multiple times.  NONE of these are acknowledged by the PLM.  I don't understand why they are not acknowledged - they appear to be valid commands.  I also don't understand why you would turn that many scenes off in rapid succession, mutiple times.  Again, because they happened to quickly, these are probably coming from a program.

Very ODD to say the least.

Section 4 shows a device direct command requesting that the device BEEP.  It's very normal with both a PLM and Device Acknowledge.

 

Conclusion:  This does not appear to be a classic All-On event.  A all on event is purportedly caused by a command collision at the PLM.  Because the event is the result of a collision,  the ISY DOES NOT HAVE KNOWLEDGE of the event.  Your logs show that the EISY does have knowledge of the event and is trying to reflect that status of the devices.

Since the event appears program related I would start by inspecting/disabling programs that work with Group 25 (On/OFF in Section 1). 

You may also want to inspect the program(s) that are issuing the multiple group OFF commands in Section 3.  I don't understand why, but the PLM is not responding to these commands.

 
 
Section 1 - Group 25 turned on then immediately turned off  
Fri 07/26/2024 12:51:42 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 11 00 Group 25 On
Fri 07/26/2024 12:51:42 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off
Fri 07/26/2024 12:51:42 PM : [INST-ACK    ] 02 62 00.00.25 CF 11 00 06          LTONRR (00) PLM Ack
Fri 07/26/2024 12:51:42 PM : [INST-ACK    ] 02 62 00.00.25 CF 13 00 06          LTOFFRR(00) PLM ACK
   
Section 2 - ISY bookkeeps the on/off status of 65 unique devices + secondary buttons  
Fri 07/26/2024 12:51:42 PM : [55 FF B6 6      ]       ST 255 (uom=100 prec=0) ISY Book keeping ON/OFF 
Fri 07/26/2024 12:51:42 PM : [55 FF B6 6      ]       ST   0 (uom=100 prec=0) 65 Unique devices
Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       RR  28 (uom=25 prec=0)  
Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       OL 255 (uom=100 prec=0)  
Fri 07/26/2024 12:51:59 PM : [50 AD 58 1      ]       ST 255 (uom=100 prec=0)  
   
Section 3 - Multiple scenes turned off - PLM doesn not respond to these commands  
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 3A CF 13 00 Group 3A Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 42 CF 13 00 Group 42 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 43 CF 13 00 Group 43 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 41 CF 13 00 Group 41 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 41 CF 13 00 Group 41 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 42 CF 13 00 Group 42 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 43 CF 13 00 Group 43 Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off 
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 4B CF 13 00 Group 4B Off
Fri 07/26/2024 12:51:59 PM : [INST-TX-I1  ] 02 62 00 00 25 CF 13 00 Group 25 Off 
   
Section 4 ? Odd beep command to device 55.FF.B6 both PLM and Device Ack  
Fri 07/26/2024 12:51:59 PM : [INST-ACK    ] 02 62 55.FF.B6 0F 30 00 06          BEEP   (00)  
Fri 07/26/2024 12:51:59 PM : [INST-SRX    ] 02 50 55.FF.B6 43.6E.51 2F 30 00    BEEP   (00)  
Fri 07/26/2024 12:51:59 PM : [Std-Direct Ack] 55.FF.B6-->ISY/PLM Group=0, Max Hops=3, Hops Left=3  

Group 25 Devices - Does not include sub-buttons

1 14 72 B6
2 14 7B DF
3 15 C7 F5
4 16 9E 2F
5 16 9E E 
6 16 A0 74
7 16 A2 45
8 16 C5 51
9 16 C7 9B
10 16 CA DA
11 16 CC 2A
12 16 CC 31
13 16 CC E 
14 16 CD 0 
15 16 CD D9
16 18 0 1E 
17 18 9F 2E
18 19 CE BC
19 1A 1E BD
20 1A 21 CA
21 1A 27 8A
22 1A 53 6A
23 1A 5F DC
24 1A AC 83
25 1A AF 70
26 1F 73 67
27 20 E7 6D
28 20 EA 46
29 24 6B C1
30 24 96 D7
31 25 4E C2
32 25 5B FA
33 25 B8 99
34 27 6E 30
35 27 70 AE
36 27 70 D6
37 29 40 87
38 2B 58 31
39 2B 5C F0
40 2B 65 B4
41 2B 6E 7E
42 3A FB D4
43 3D F4 FF
44 3D F9 1B
45 3D FE D6
46 3E 10 47
47 3E 8 E1 
48 3E E EF 
49 4B DE 4E
50 4C 85 3D
51 4E 27 F9
52 4E 29 6F
53 4F A0 20
54 50 AD 58
55 50 B0 CA
56 50 B9 76
57 51 74 E5
58 52 E0 A6
59 53 28 E5
60 53 29 8B
61 54 55 53
62 54 57 A3
63 54 E3 2F
64 55 FF B6
65 70 7B 5B

 

  • Thanks 1
Link to comment

@IndyMike Thanks SO much for helping decipher the log! I'm on vacation for the week so I'll get back to this when I get home. I will say this, since replacing the batteries in the one suspect motion sensor, the problem has not reoccurred.

Link to comment

Apparently the motion sensor was not the issue, 1:52 a.m. everything triggered again last night. Back to the drawing board. Just so odd that it's always the same time every night. I'll have to review all my programs and see if I have anything scheduled to run at that time, but I'm pretty sure I don't.

Link to comment
32 minutes ago, arzoo said:

Apparently the motion sensor was not the issue, 1:52 a.m. everything triggered again last night. Back to the drawing board. Just so odd that it's always the same time every night. I'll have to review all my programs and see if I have anything scheduled to run at that time, but I'm pretty sure I don't.

That's unfortunate.  Do you have any programs that re-trigger on an hourly basis?  The start times can drift if delayed by other events or if they are qualified by sunrise/sunset.  I've seen programs "drift together" and cause issues.

I have a number of programs that Poll devices throughout the day -

Flood Night Poll - [ID 006E][Parent 0002][Run At Startup]

If
        From    Sunset  +  1 minute
        To      Sunrise - 10 minutes (next day)
    And Time is Last Run Time for 'Flood Night Poll' + 15 minutes
 
Then
        Send X10 'F1/Status Request (10)'
        Wait  9 seconds
        Send X10 'F3/Status Request (10)'
        Wait  9 seconds
        Send X10 'F5/Status Request (10)'
 
Else
   - No Actions - (To add one, press 'Action')
 

 

  • Thanks 1
Link to comment

@arzoo you could use the raw text portion of the find/replace option in the programs to potentially find a time reference. Or I would suggest you copy the root folder to the clipboard then paste in a text editor and search for time or other triggers in the text of all the programs. (right click on the root folder and select COPY folder to clipboard - NOT "export")

Have you checked the summary tab for the programs? Anything showing "last run" at the time this happens? You originally said it was happening at random hours so maybe you'll see it during the day rather than 1am time. 

 

  • Like 2
Link to comment

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

Link to comment
28 minutes ago, arzoo said:

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

There's a lot of symptoms and great analysis in this post... but if it is a flaky Insteon MS, I've found that letting the battery run down can confuse the MS (at least the MS1) and cause unpredictable behavior. I made it a practice when I had them to factory reset and "restore device" when replacing a low/depleted battery. I got many years with them when staying ahead of the battery replacements, in fact sold them with the house.

  • Like 3
Link to comment
4 hours ago, arzoo said:

I'm back to thinking the one motion sensor is the culprit. Replacing the battery did not help, but I think it's actually malfunctioning/faulty. With that motion sensor disabled, the issue has not reoccurred for at least one day. I'll give it a few more, and then replace it with a new one.

Can you provide the Insteon address for the motion sensor?  I can check your event log to see if it sent something odd.

I agree with @paulbates regarding the factory reset of the MS1's.  This has become standard practice for me whenever I replace a battery.

  • Like 1
Link to comment
On 8/2/2024 at 2:25 PM, IndyMike said:

Can you provide the Insteon address for the motion sensor?  I can check your event log to see if it sent something odd.

14 72 B6 1

But the issue continues even with the sensor disabled. I'll have to try some of the other suggestions to continue debugging. I'll focus on the fact that it only happens at 52 minutes past the hour.

Link to comment

Your motion sensor (14 72 B6) appears in almost every log entry, so chances are this is the device that's causing you problems.  Disabling the device does not stop it from sending spurious data into the Insteon network. You need to remove its battery in order to completely disable it.

You should also look for a program that may be running around the time of your issues

Edited by Techman
Link to comment
3 hours ago, Techman said:

Your motion sensor (14 72 B6) appears in almost every log entry, so chances are this is the device that's causing you problems.  Disabling the device does not stop it from sending spurious data into the Insteon network. You need to remove its battery in order to completely disable it.

You should also look for a program that may be running around the time of your issues

That's good to hear and will do when we get home! Thanks

Link to comment
On 8/3/2024 at 3:05 PM, arzoo said:

14 72 B6 1

But the issue continues even with the sensor disabled. I'll have to try some of the other suggestions to continue debugging. I'll focus on the fact that it only happens at 52 minutes past the hour.

Thanks - that helps.

As @Techman indicated, the 14.72.B6 sensor shows up a lot.  It appears to have an on/off time of 30 seconds.  It is being evaluated by  8 separate programs.  One (or two) of those programs is turning on scene 25 and then immediately turning it off again.  Some of these may be disabled - they will still show up in the log.

There appears to be a second sensor (14.7B.DF) that is also activating.  This one appears to have an on/off interval of 1 minute.  It is also being evaluated in 8 separate programs with two shared programs (highlighted green below).  Again, there is a program turning on scene 25 and the immediately turning it off again.

There is a 3rd device that is turning on/off multiple scenes.  I can't tell the device type.  The device is evaluated by 6 different programs. 

All three of the above devices were active at the 12:51:59 mark when things went out the window.  You should be able to search for programs where these devices are used by using the "find" feature as @Geddy described.  I have a feeling you have multiple programs firing at the same time and causing issues.

Motion sensors are notorious for firing multiple communications per event.  Both of your devices send 2 On messages + 1 cleanup per event.  As a precaution, you may want to add a wait to your programs to prevent collisions. 

I still do not see a smoking gun anywhere that would indicate what triggered the "event".  Try some precautionary changes and things may become clearer.  This could also be caused by a non-insteon event (PG3, rest interface, Home Assistant) that would not show up in the Event Viewer.

 

image.thumb.png.f3bc3a8340d0d8fd90affccb29f6387e.png

 

image.png.cef7147b1b83f6d9f5fdd636a21e0ad6.png

 

image.thumb.png.f93d773d93b3a5e9ce97ca8af9b183d2.png

 

image.png.9a34bda56465cef468391b41ec79f1a4.png

 

image.png.4cc8c8f2e7705bacf53a0843c353f853.png

 

  • Like 1
Link to comment

I removed the battery from the suspected sensor (14 72 B6 1) but the problem triggered again last night at 12:52. The fact that it's always around the same time has me thinking what programming I might have in place that's the culprit. The only programs I'm running with an hourly trigger are for monitoring the heartbeat signal on four Insteon water sensors. When the water sensors are working correctly, they send a heartbeat once about every 24 hours. I have programs that check every hour, incrementing a counter. If the counter exceeds 24 then I assume the battery is low. Anyway, I'll disable those programs and see if it makes any difference.

Link to comment
Guest
This topic is now closed to further replies.

×
×
  • Create New...