Intermittent program execution delays

[AKAPADIA]

I recently moved to Polisy from ISY. I am running IoP 5.4.3. Things are mostly working correctly, except occasionally I have large (40+ second) lags between the trigger of a program and the eventual observed effect of its execution. 

For instance, in my family room I have an occupancy sensor and a motion sensor tied to my Elk system which I use to trigger lights on using an Isy program. What used to take ~2 seconds to trigger on the Isy just took 40+ seconds on IoP. 

When I look at the debug logs, I do see the same 40 second delay (see screen shot of log file) - between the time the IsY saw the occupancy sensor trigger and by the time the light came on:

When I look at the error logs in between those two events, I don't see anything amiss. How would I debug this further?

Tue 2022/04/12 07:48:06 PM    0    -170001    <s:Envelope><s:Body><u:Unsubscribe xmlns:u="urn:udi-com:service:X_Polisy_Service:1"><SID>uuid:27</SID></u:Unsubscribe></s:Body></s:Envelope>
Tue 2022/04/12 07:49:09 PM    0    -170001    <s:Envelope><s:Body><u:Subscribe xmlns:u="urn:udi-com:service:X_Insteon_Lighting_Service:1"><reportURL>REUSE_SOCKET</reportURL><duration>infinite</duration></u:Subscribe></s:Body></s:Envelope>

Tue 2022/04/12 07:49:18 PM    0    -170001    [TCP-Conn] -1/-140002, Net Module Rule: 17
Tue 2022/04/12 07:49:18 PM    0    -170001    [TCP-Conn] -2/-140002, Net Module Rule: 16
Tue 2022/04/12 07:49:18 PM    0    -170001    [TCP-Conn] -2/-140002, Net Module Rule: 15
Tue 2022/04/12 07:50:42 PM    0    -170001    <s:Envelope><s:Body><u:Unsubscribe xmlns:u="urn:udicom:service:X_Insteon_Lighting_Service:1"><SID>uuid:28</SID></u:Unsubscribe></s:Body></s:Envelope>

I wondered if the IsY was too busy - because there is pretty heavy logging - lots of things happening with the Elk Node server. I wondered if it was related to that Insteon device, but when I use my UD Mobile app to remotely operate the light, it's instantaneous. 

Edited April 13, 2022 by arshishk
More information

[AKAPADIA]

More info:

Other programs triggered by the same keypad also exhibit delays. However, this is intermittent. Sometimes it works fine. And sometimes there are delays. It almost seems like there is a warm up where if I trigger this device 'cold' there is a delay, but subsequent interactions allow the program to run quickly. Very puzzling.

This cannot be Insteon related because it works fine with the old Isy - the problem shows just in the Polisy setup.

Edited April 13, 2022 by arshishk

[AKAPADIA]

Text of program to turn on light attached!

[lilyoyo1]

My polisy has been slower than my Isy stuff. With this still being alpha/beta, I've chalked it up to that. 

They're times when things run extremely fast and others extremely slow. I think it will get better in time. 

One thing I did notice is with the PLM USB stick. Communication from devices is much slower along with missed signals. 

[Michel Kohanim]

@arshishk,

What exactly is Family Flood.1?

@lilyoyo1, 

Please post programs that are intermittently slow.

For those with these issues, there has been a major change but only for network resources: now, network resources and node servers are not queued. This means that if you have programs sending network commands, and with default connect time out of 10 seconds, there might be delays. Thus my question about Flood.1.

With kind regards,
Michel

[lilyoyo1]

27 minutes ago, Michel Kohanim said:

@arshishk,

What exactly is Family Flood.1?

@lilyoyo1, 

Please post programs that are intermittently slow.

For those with these issues, there has been a major change but only for network resources: now, network resources and node servers are not queued. This means that if you have programs sending network commands, and with default connect time out of 10 seconds, there might be delays. Thus my question about Flood.1.

With kind regards,
Michel

It's really nothing major. Just basic if sensor is switched on turn lights on. I use zwave sensors (dome) to an insteon switch. There's been no rhyme or reason for the slow down. Sometimes it's so fast I'm amazed and others I've turned the light on as the program starts to run.

 

[AKAPADIA]

@Michel Kohanim, Family Flood is an Insteon 8 button keypad's 'main' button (top left). I have another interesting observation to share from yesterday. I looked at the ISY UI while I got my wife to trigger the sensor in a repro of the issue. The ISY UI did not show the program as triggering for a full 20+ seconds after the sensor showed that it had been triggered (the ELK node output zone swung to "On"). Once the program triggered (icon turned green), the light came on instantly.

Also noticing this problem is more widespread and affects other programs as well. Appears random.

Note: I am using a wired PLM 2413U (brand new)

Edited April 13, 2022 by arshishk

[vbPhil]

Yes I agree that Polisy/Polyglot/IoP might still need some hardening. I have a small isy utility program that increments a few variables every night a 11:59. It's been running error free for many years. Last night it didn't increment any of the variables. Nothing in the logs that I can see except that at 11:59 there was activity between the PG3 WirelessTag Node Server and ISY. I'll keep an eye on this for anything else that I might notice.

Of note, I ran the program manually before checking the ISY program summary readout so I can't say if the program actually ran last night like it should have.

[Michel Kohanim]

Thank you all for your feedback. We are trying to reproduce ... 

@arshishk, did the event viewer show DON or did you get DON after 20 seconds? 

@vbphil, are these variables incremented based on INSTEON events

Do all of you have 2413U?

With kind regards,
Michel

[lilyoyo1]

3 minutes ago, Michel Kohanim said:

Thank you all for your feedback. We are trying to reproduce ... 

@arshishk, did the event viewer show DON or did you get DON after 20 seconds? 

@vbphil, are these variables incremented based on INSTEON events

Do all of you have 2413U?

With kind regards,
Michel

I'm using 2413s. The USB stick had too many issues for me

[Michel Kohanim]

@lilyoyo1, thank you. So, you still have the delays even with 2413S?

With kind regards,
Michel

[lilyoyo1]

Just now, Michel Kohanim said:

@lilyoyo1, thank you. So, you still have the delays even with 2413S?

With kind regards,
Michel

Yes. Its a night and day difference between the 2. The USB stick was noticeably slower than the PLM and would miss button presses etc.  I havent tried the USB stick with the latest firmware yet

With the 2413s PLM, 90-95% of the time everything runs perfect. Then out the blue I'll get slowdowns. The isy is receiving the commands. Just execution becomes slow

[vbPhil]

16 minutes ago, Michel Kohanim said:

@vbphil, are these variables incremented based on INSTEON events

The variables are not based on Insteon events. This is the code for one of them.

Quote

DayFlip2 - [ID 0049][Parent 0025][Not Enabled]

If
        $iDay2 < 1
 
Then
        $iDay2 += 1
 
Else
        $iDay2  = 0
 
iDay2 values are 0 and 1
 

I'm running the Serial PLM. 

To explain more. This program is scheduled to run at 11:59 and is enabled. It kicks off 3 other program that each increment their own variable.

Quote

EndOfDay - [ID 0026][Parent 0025]

If
        Time is 11:59:00PM
 
Then
        Run Program 'DayFlip2' (If)
        Run Program 'DayFlip3' (If)
        Run Program 'DayFlip4' (If)
 
Else
   - No Actions - (To add one, press 'Action')
 

 

Edited April 14, 2022 by vbphil
added more clarity

[AKAPADIA]

Hi @Michel Kohanim, 

The event viewer does not explicitly show DON; it just says the light went to 100%. I think that corresponds to DON, though. I don't think this is related to the PLM at all, because I have yet another very interesting observation to report:

In short:

- I went to my family room and pressed keypad button F to operate a blind. Nothing happened

- I walked to my powder bath and the motion sensor there triggered, but the bath light did not come on

- I started counting the seconds in my head and, after ~30-40 seconds, both the blind went up and powder light came on virtually simultaneously. I am betting if I had been watching the Isy UI, both programs had not run until this point - inspite of the trigger conditions being received successfully. It's like Isy had suspended execution of all programs for some reason. 

- The log files show exactly this result. Highlighted in yellow, you see the effect of the blind activation thru keypad F (and the eventual blind execution thru the toggle of keypad E from on->off). Highlighted in green, you see the effect of the powder motion sensor triggering and then the eventual powder light coming on at the same time the blind program triggered. 

Edited April 14, 2022 by arshishk

[MrBill]

This seems very much like Noise on the powerline delaying Insteon signals. 

https://wiki.universal-devices.com/index.php?title=INSTEON_Signal_/_Noise_Troubleshooting

Open the Event Viewer and set the dropdown to level 3 and capture what's happening there.  It's more detailed with transmission attempts than the log.

 

[AKAPADIA]

Thanks @MrBill. This only started happening after my change to Polisy. Also, in the previous post dated 4/13, 11:19, note that the program didn't show as running in the Isy admin UI until 10s of seconds after the motion sensor triggered. This seems to have nothing to do with Insteon and rules it out from my perspective, unless I am missing something. If the program had shown itself in the 'running' state on trigger, and remained running for 20+ seconds, and then the light came on, then you might conceivably say Insteon communication is the cause.

Edited April 14, 2022 by arshishk

[MrBill]

43 minutes ago, arshishk said:

Thanks @MrBill. This only started happening after my change to Polisy. Also, in the previous post dated 4/13, 11:19, note that the program didn't show as running in the Isy admin UI until 10s of seconds after the motion sensor triggered. This seems to have nothing to do with Insteon and rules it out from my perspective, unless I am missing something. If the program had shown itself in the 'running' state on trigger, and remained running for 20+ seconds, and then the light came on, then you might conceivably say Insteon communication is the cause.

If you're unwilling to check Level 3 Event Log then I wish you luck, you know best!

[AKAPADIA]

I have the event log open and capturing traffic at level 3 and I am trying to reproduce the problem. I was merely providing you with extra information to help narrow the problem down faster, I certainly do not claim to know best :-). Thank you for your help and input. 

[AKAPADIA]

Reporting on event log level 3 with the same repro scenario:

- operate Blind (this is button 5 on keypad with address 2B.A8.02 at the start of the log. It shows up at time 2:33.30 in the level 3 log but at time 2:33.57 in the Excel based log)

- immediate walk to Powder Bath and trigger light few seconds later (this is shows up at time 2:34.02 in the level 3 log, but at time 2:34.09 in the Excel log. The interesting thing here is that it takes me only a few seconds to walk over to the bathroom but the event log shows a much larger time difference)

- wait ~40 seconds

- observe both blind go down and light come on in powder bath, virtually immediately after each other (the powder bath light switch address is 2B.69.8D and can be seen being activated in the level 3 log  2:34.03, and the program to lower the blind sets a variable on response to the key pad press, which can be seen at 2:34:03 (variable # 50 set to 255))

 

Level3log.txt

[AKAPADIA]

@Michel Kohanim, I think I have the cause of this one figured out - though I cannot explain why it happens. I had a program that would refresh ELK zone voltages every 3 minutes (attached). It made 3 sequential network calls. Since the old ISY is no longer at that IP address (I'm now running on Polisy), these calls should timeout. They have an explicit timeout of 1 second. So I see no obvious reason for causing the program execution delay. However, I have proved that disabling this program (which is fine because I no longer need it; the ELK Node server keeps voltages up to date by itself) eliminates all the delays. Reenabling the program with no other changes reintroduces the delays, so I know this is the root cause.

Two attachments: the errant program and the network resource definitions.

[macjeff]

I was using a Variable to make alexa announce and to change the status of a keypadlinc button.  To take variables out of the equation I used a Node server Called Virtual Buttons that basically pushes a button to make alexa do something.  If I push the button in the Node server its instant.

This AM we closed a garage door and also had motion in the driveway.  Two separate devices with two separate programs.  Both used the new virtual buttons.  Both light up a keypad button.

With both of them the device was triggered.  Alexa said nothing and the keypadlinc button did not light up.

About 5 min later Alexa said something and I looked and the keypadlinc was lit up.

for the motion it worked once right away but a few min later I see the device trigger again (someone walked past the motion) in the ISY but the program has yet to run at all.

So its a delay and sometimes a total ignore for me.

Makes me wonder what else is not working.  Like I have leak sensors.  What if there is a leak.  Will it let me know?

 

[Michel Kohanim]

18 hours ago, arshishk said:

I think I have the cause of this one figured out - though I cannot explain why it happens

@arshishk, thank YOU so very much. I can explain why: in ISY, network resources were put in a queue and another thread would run them. In IoP, they are inline. THANK YOU.

With kind regards,
Michel

[macjeff]

9 hours ago, Michel Kohanim said:

@arshishk, thank YOU so very much. I can explain why: in ISY, network resources were put in a queue and another thread would run them. In IoP, they are inline. THANK YOU.

With kind regards,
Michel

TEAMWORK!!!  I wish the US Gov could work together so well and get things fixed!!!!

[vbPhil]

12 hours ago, Michel Kohanim said:

@arshishk, thank YOU so very much. I can explain why: in ISY, network resources were put in a queue and another thread would run them. In IoP, they are inline. THANK YOU.

With kind regards,
Michel

@Michel KohanimSo I'm just wondering, in IoP are the network resources going to stay "inline" or will you change it to how the ISY performed?

[AKAPADIA]

@Michel Kohanim, but why would three network calls with a 1 second timeout introduce delays of up to 30-40 seconds? That's the part that I found unintuitive.