ISY delays in rest calls

[Sirmeili]

I am running a polisy with an integration into home assistant (HA) . I've noticed in HA when I add an insteon device to an automation triggered by another isy device, then the control often has a 2 second delay in the rest response from the ISY.

I can no say for sure that this is on the ISY side at all, but I'm just covering my bases here.

Is there anything on the ISY to prevent throttling on the rest API or that would cause a 1-2s delay right after an outgoing message?

For instance, as a test when my utility closet opens or closes, I trigger the light in that room to turn on/off (i know this can be done solely in the ISY, but this is merely a test to isolate the issue to the ISY or it's integration in HA).

When the automation is triggered by the isy event, there is a 1-2 second delay in the rest call to the ISY (call and response denoted by the lines surrounded by **)

2023-07-29 22:12:30.246 DEBUG (MainThread) [homeassistant.components.isy994] Sensor None turning On via the Primary node sending a DON command
2023-07-29 22:12:30.248 DEBUG (MainThread) [homeassistant.components.isy994] Heartbeat timer starting. Now: 2023-07-30 02:12:30.247908+00:00 Then: 2023-07-31 03:12:30.247869+00:00
2023-07-29 22:12:30.248 DEBUG (MainThread) [pyisy] ISY Node Control Event: NodeProperty(control='DON', value=1, prec='0', uom='', formatted=None, address='32 86 20 1')
2023-07-29 22:12:30.249 DEBUG (MainThread) [pyisy] ISY Updated Node: 32 86 20 1
**2023-07-29 22:12:30.262 DEBUG (MainThread) [pyisy] ISY Request: http://10.0.0.172:8080/rest/nodes/50%20D1%207F%201/cmd/DON**
**2023-07-29 22:12:31.873 DEBUG (MainThread) [pyisy] ISY Response Received: /nodes/50%20D1%207F%201/cmd/DON**
2023-07-29 22:12:31.873 DEBUG (MainThread) [pyisy] ISY command on sent to 50 D1 7F 1.
2023-07-29 22:12:32.186 DEBUG (MainThread) [pyisy] ISY Updated Node: 50 D1 7F 1

If I call the automation directly (not triggered by a ISY event) the call to turn on/off the light is not delayed, it's almost instantaneous (call and response denoted by the lines surrounded by **)

**2023-07-29 22:21:17.173 DEBUG (MainThread) [pyisy] ISY Request: http://10.0.0.172:8080/rest/nodes/50%20D1%207F%201/cmd/DON**
**2023-07-29 22:21:17.181 DEBUG (MainThread) [pyisy] ISY Response Received: /nodes/50%20D1%207F%201/cmd/DON**
2023-07-29 22:21:17.182 DEBUG (MainThread) [pyisy] ISY command on sent to 50 D1 7F 1.
2023-07-29 22:21:17.496 DEBUG (MainThread) [pyisy] ISY Updated Node: 50 D1 7F 1

Again, I'm not saying it's the ISY or the Polisy, but just covering my bases.


I have run tests where I just turn on/off a device from HA in rapid succession and am not able to reproduce the delay there either, making me think it's HA, but again just checking.

Is there anywhere in the ISY/Polisy to see the Rest API call logs? I would like to see if the outgoing response matches the timestamp of the one in HA. My working theory is that HA is for some reason throttling it, but I can't say this for sure without logs on the ISY side.

Edited July 30, 2023 by Sirmeili

[mmb]

@Sirmeili I use Fast Off on an Insteon switch to initiate a REST command to my alarm system to arm/disarm.

In my case the Rest almost instant.

Fyi I did have a problem with my Polisy and Rest after migration from the ISY.  The Rest wasn't working at all, but was able to track it down to timeout.  Increasing the timeout on the command fixed it.

The Event Viewer on level 3 helped sort out the problem.

 

 

[Sirmeili]

1 hour ago, mmb said:

@Sirmeili I use Fast Off on an Insteon switch to initiate a REST command to my alarm system to arm/disarm.

In my case the Rest almost instant.

Fyi I did have a problem with my Polisy and Rest after migration from the ISY.  The Rest wasn't working at all, but was able to track it down to timeout.  Increasing the timeout on the command fixed it.

The Event Viewer on level 3 helped sort out the problem.

 

 

I will check that out.

The ISY setup is new. I just migrated from HomeSeer back to the ISY (and into Home Assistant). To be clear, the rest calls manually triggered in HA are instant. They happen in about 80ms. It's whenever an Automation is triggered in HA where there is an action calling back to the ISY. For instance, I have some rooms where if you  fast off the main light, it turns off the TVs, lights, etc. I like to keep all that pretty much in the same place for easier maintenance. So as part of those automations I call a scene in the ISY to turn off all the Insteon stuff (again, I know I COULD do this part in the ISY, but I like it all in one place). 

Calling that scene manually from HA is instant, but when the automation calls it based on the fast off trigger, it is delayed 1.5-2 seconds. Every. Single. Time. Again, I don't know if it's HA, the integraiton or the ISY.

I will check out the level 3 event viewer  Thanks!

[Sirmeili]

So I was able to reproduce it manually. I do think now it's something on the ISY, but this is just sort of unacceptable. it should be able to handle more than requests at a time like that. I would assume it is multi threaded, but perhaps not.

it tried calling the ISY rest end points directly and yup, even subsequent calls of the exact same are immediate for the first call and then about a 1.5 second delay for one immediately after. My guess is that the ISY is somehow throttling outgoing/incoming requests. This is bad because 1.5s may not seem like a lot, but when you open a door it can feel like forever when waiting for a light to turn on.

I didn't have this issue when I had insteon directly set up in HomeSeer, and I honestly don't recall this an issue when I was using the ISY plugin for Homeseer a few years ago, but I was on an ISY 994 back then, not a polisy.

My last test was just turning on 2 light and then turning them off (4 actions) and yup, the first one was fast, but the next 3 commands all took 1 second each. Which again, doesn't seem like that much, but in the world of networking and ms fast transactions, 1000ms is a long time.

I will have to see if  I can find another option because these delays are not acceptable unless someone can tell me how to speed them up?

 

[bpwwer]

If you can put together a test case that shows this and allows someone else to easily reproduce it, submit a ticket as something seems wrong.

With PG3, we sometimes send many requests a second to the IoX with response times in the 10-20ms range or less).  If there were 1 second delays, some operations could end up taking a very long time.

[Sirmeili]

Just a quick note that it appears even from the admin screen you are limited to about 1 command a second.

I created a program which turns a light on/off 2 times and it doesn't seem to have such a limitation, so the limitation definitely is not from the ISY to the PLM, it's just in it's incoming/outgoing messaging (don't know if the Admin uses the same REST interface or what) to UIs/Integrations that seem to have a limitation.

[Sirmeili]

1 minute ago, bpwwer said:

If you can put together a test case that shows this and allows someone else to easily reproduce it, submit a ticket as something seems wrong.

With PG3, we sometimes send many requests a second to the IoX with response times in the 10-20ms range or less).  If there were 1 second delays, some operations could end up taking a very long time.

Ok so I might be wrong here. 

I opened up the event viewer, opened a tab pointing to a rest endpoint (http://10.0.0.172:8080/rest/nodes/50%20D0%20C4%201/cmd/DOF) and just hammered the refresh. I can see all the rest calls come in in quick succession. I don't have a quick way to test if they are in fact sending commands to the device (since it's the same value).

I'll have to see if there is a way for me to call the rest API with subsequent calls with alternating values.

[Sirmeili]

Ok. I guess I will backtrack on this somewhat. If calling from the ISY (I put a simple html page on the isy with JS to call the same device in rapid succession) it does seem to operate fast (most the time). I do get an occassional 2 second delay on one of the requests. This is if I call the 4 requests in async (though they do still execute in order).

In this example you can see the 4th request still took some time but the first 3 completed in under 50ms.

2023-07-30T20:18:32.500Z Recived:
2023-07-30T20:18:30.641Z Recived:
2023-07-30T20:18:30.640Z Recived:
2023-07-30T20:18:30.640Z Recived:
2023-07-30T20:18:30.597Z Sent:
2023-07-30T20:18:30.597Z Sent:
2023-07-30T20:18:30.596Z Sent:
2023-07-30T20:18:30.596Z Sent:

Even if I do it serially, they appear to execute quite well to a point. The third and forth are delayed by almost 1.5s and 1s respectively.
2023-07-30T20:26:17.993Z Recived:
2023-07-30T20:26:16.983Z Sent:
2023-07-30T20:26:16.982Z Recived:
2023-07-30T20:26:15.566Z Sent:
2023-07-30T20:26:15.566Z Recived:
2023-07-30T20:26:15.559Z Sent:
2023-07-30T20:26:15.559Z Recived:
2023-07-30T20:26:15.552Z Sent:

Which I guess is to be somewhat expected.

I'm still confused that in the scenario of a sensor has gone off, notified HA, and HA turned around and called back the ISY, why it is taking 1.5 seconds (consistantly). 

Here are some tests where I manually triggered a door sensor and ran my script immediately after. The first request after doing so is delayed, again by about 1.5s.:
2023-07-30T20:36:49.935Z Recived
2023-07-30T20:36:49.929Z Sent
2023-07-30T20:36:49.929Z Recived
2023-07-30T20:36:49.057Z Sent
2023-07-30T20:36:49.057Z Recived
2023-07-30T20:36:48.174Z Sent
2023-07-30T20:36:48.174Z Recived
2023-07-30T20:36:46.664Z Sent

Could the ISY really be tied up that much after a sensor has been tripped?

I tried the same scenario with an ISY program. Door triggers program, program turns on light. Same issue. there is a good 1.5s delay, which makes little sense to me. I've attached the log from the ISY for just this test in case anyone can see what is going on. I don't know why there are a lot of "Previous Message Ignored"  like things are being sent multiple times.
 

To be clear here, I'm not trying to hammer the ISY with requests. My simple use case is a sensor is tripped, sent to HA, HA sends a request back, as explained above, this even happens with a program on the ISY.

My ultimate use case is to have this automation only run under certain scenarios, which I can't really do with an Insteon Scene (I.e., only turn on the light if the door opens and X is true).

*note my tests above are now only limted to the ISY showing that at times, it is in fact taking 1 to 1.5s to process a rest request).

 

ISY-Events-Log.v5.6.3__Sun 2023.07.30 04.50.20 PM.txt

[bpwwer]

I wrote a python script to quickly send a number of requests to the IoX and log the response times.

import requests

cmd = 'DON/100'
for i in range(0,25):
    r = requests.get('http://192.168.92.11:8080/rest/nodes/17%20D%2096%201/cmd/{}'.format(cmd), auth=('admin', 'admin'))
    print('{}: {}: {}'.format(i, cmd, r.elapsed.total_seconds()))
    if cmd == 'DON/100':
        cmd = 'DOF'
    else:
        cmd = 'DON/100'

When I send a command to a node server node, all the response times are short,  ~30ms.   However, when I alternate DON/DOF to an Insteon device (above was a micro dimmer I believe) the results tend to be much more varied. In one run it ranged from ~1ms to 1.5 seconds.  So this seems to correspond to what you're seeing.

My theory is this:

With the node server command, there's no waiting for anything.  IoX receives the request, sends it to the node server and sends back a response.  The only delay is the time it takes to send the command to the node server via PG3.

With the Insteon device, the IoX has to wait for status response from the device before it sends the response back, hence the time to get a response depends largely on the Insteon device communication.  I.E. Iox receives the request, it sends the cmd to the Insteon device, it waits for the Insteon device to respond and finally sends the response back.

I think what you might be seeing is a slowdown in the Insteon communication when requests are sent quickly.  To test this, I added a delay in my loop of 15 seconds between sending commands and then all the responses are about 2ms. 

Again, this seems to correspond to what you're seeing with the automation.  Insteon traffic triggers the automation which then sends an Insteon command immediately, but you see the delay, triggering manually (meaning there's no initial Insteon traffic) has no delay.

I believe what you're seeing is normal.  Another test you could do is to use an ISY program to trigger the light based on the motion sensor (or whatever triggering device you're using).  I think you'll see the same delayed response.   Another test would be to set the triggering device and the switch in a Insteon scene so it triggers directly.  Here you should see much less delay as the trigger device immediately sends the command to the switch. 

[Sirmeili]

I will try that out. My only thing is when I had this same setup in homeseer with it controlling the insteon devices directly, I didn't have any such delay.

 

Again, the delay I'm seeing is after a sensor is updated, the subsequent command is delayed. My tests tried be catching requests out and that actually seemed to work ok. I don't get why the sensor coming in is slowing the next response. I also don't get all the "previous message ignored" logs which seems to fill the 1.5s gap.

Edited July 30, 2023 by Sirmeili

[Sirmeili]

14 hours ago, Sirmeili said:

I will try that out. My only thing is when I had this same setup in homeseer with it controlling the insteon devices directly, I didn't have any such delay.

 

Again, the delay I'm seeing is after a sensor is updated, the subsequent command is delayed. My tests tried be catching requests out and that actually seemed to work ok. I don't get why the sensor coming in is slowing the next response. I also don't get all the "previous message ignored" logs which seems to fill the 1.5s gap.

I should be more clear. Anytime the ISy gets an update from a device controlled outside the isy, I get this. I.e. if I use "Fast off" on a switch/dimmer, I get the same delay on any subsequent actions.

[MrBill]

tagging @shbatm on this discussion because it involves Home Assistant, even tho you've manually reproduced and indicated the issue is on the IOX/ISY side.

[Sirmeili]

12 minutes ago, MrBill said:

tagging @shbatm on this discussion because it involves Home Assistant, even tho you've manually reproduced and indicated the issue is on the IOX/ISY side.

I've been talking with @shbatm on the PyISY github as well. I don't think it will be much help here. It seems to be a limitation of the ISY.

I'm looking at using the Insteon HUB directly with HA to see if I have that same delay. I liked the ISY for specific things, but I can't live with that 1-1.5s delay between a sensor/device sending an update and it allowing a command to another device (again, I didn't have this limitation when using HomeSeer's direct insteon control)

[bpwwer]

If you have a reproducible test case that shows the issue, submit a ticket.

My opinion is that what you are seeing is normal.  Likely because the IoX is prioritizing reliability over performance on the Insteon network.  The HomeSeer plugin may be doing the opposite.   It's also possible that something has changed in your Insteon communications that has made device communication less reliable.  The Insteon protocol has redundancy built into it so that it can work through many issues, but that can also cause reduced performance.

Another possibility is that the HomeSeer plugin was optimizing things by automatically creating a "scene" for cases where you were setting up a sensor to trigger another device.   The IoX/ISY won't do that, you have to create the scene and add the devices yourself. 

Have you tried configuring things as I suggested above?  Those two tests would help determine if what you're seeing is normal Insteon communication delays or not and also help to eliminate HA as contributing to the issue (which I don't think it is).

[Sirmeili]

1 hour ago, bpwwer said:

If you have a reproducible test case that shows the issue, submit a ticket.

My opinion is that what you are seeing is normal.  Likely because the IoX is prioritizing reliability over performance on the Insteon network.  The HomeSeer plugin may be doing the opposite.   It's also possible that something has changed in your Insteon communications that has made device communication less reliable.  The Insteon protocol has redundancy built into it so that it can work through many issues, but that can also cause reduced performance.

Another possibility is that the HomeSeer plugin was optimizing things by automatically creating a "scene" for cases where you were setting up a sensor to trigger another device.   The IoX/ISY won't do that, you have to create the scene and add the devices yourself. 

Have you tried configuring things as I suggested above?  Those two tests would help determine if what you're seeing is normal Insteon communication delays or not and also help to eliminate HA as contributing to the issue (which I don't think it is).

I did your test of a program and saw the delay. I even posted the logs and showed how there were multiple "Previous message ignored"

I moved a few devices to HA using the built in integration (not the isy one) and I get the same issue. I guess this might explain why Insteon is dying a slow death. I never had this problem in HS and I really don't want to go back there just to get Insteon to be performant. Perhaps Mark over there wrote it to be more performant and less reliable, but it was always 100% reliable for me.

Waiting 1.5s for a "command" to go out after a signal has already been received should not be acceptable in home automation these days.

I'm not ruling out it's my environment, I just find it odd since I never saw this in HS.

[Javi]

3 hours ago, Sirmeili said:

I've been talking with @shbatm on the PyISY github as well. I don't think it will be much help here. It seems to be a limitation of the ISY.

I'm looking at using the Insteon HUB directly with HA to see if I have that same delay. I liked the ISY for specific things, but I can't live with that 1-1.5s delay between a sensor/device sending an update and it allowing a command to another device (again, I didn't have this limitation when using HomeSeer's direct insteon control)

Hi @Sirmeili, I commented on your Git issue over the weekend.  It is very likely this limitation exists in HomeSeer/Insteon-Hub  also or there would be frequent Command Drops or Status Reporting would be incorrect. 

Many situations where an Insteon device triggers another Insteon Device could be handled in Scenes, this is even faster than using network commands.  ISY allows scene adjustments in programs so you can  adjust scenes based on (IF/ELSE) conditions. 

In my experience every protocol has its advantages and disadvantages.  Insteon scenes and communication distance (due to power line) work better for me then all the other protocols.  Insteon reports status almost instantaneously where as Z-Wave can be delayed by a few seconds.  Most of the Insteon battery powered devices have been removed from my system as they had too many issues the biggest being battery life.  Z-wave battery life appears to be 5-10 times longer.  The Z-Wave battery sensors also appear to report a change in status better than the insteon sensor...unless the insteon device is still awake.

 

[Sirmeili]

3 minutes ago, Javi said:

Hi @Sirmeili, I commented on your Git issue over the weekend.  It is very likely this limitation exists in HomeSeer/Insteon-Hub  also or there would be frequent Command Drops or (B) Status Reporting would be incorrect. 

Many situations where an Insteon device triggers another Insteon Device could be handled in Scenes, this is even faster than using network commands.  ISY allows scene adjustments in programs so you can  adjust scenes based on (IF/ELSE) conditions. 

In my experience every protocol has its advantages and disadvantages.  Insteon scenes and communication distance (due to power line) work better for me then all the other protocols protocols.  Insteon reports status almost instantaneously where as Z-Wave can be delayed by a few seconds.  Most of the Insteon battery powered devices have been removed from my system as they had too many issues the biggest being battery life.  Z-wave battery life appears to be 5-10 times longer.  The Z-Wave battery sensors also appear to report a change in status better than the insteon sensor...unless the insteon device is still awake.

 

So my biggest use case for this scenario is turning off a "room". I use Fast off on the main light to trigger a scene with no controllers to turn off all the insteon devices. The problem with a 1.5s delay is that there is no instant feedback, so when this happens you always stop..wait for the scene then move on.

With the near instant reaction in HS, I didn't have this "second guessing" if a room was going to turn off (i find that Fast off is a bit fiddly from time to time).

Because of this adjusting scenes in a program doesn't help ,but I can see how it could help with potentially turning a light on/off based on the time of day.

I don't mean to sound ungrateful or anything. I think just my expectations were set by another implementation that as @bpwwer said could have been optimized for performance over reliability.

[Sirmeili]

So, QQ that I don't think I saw an answer to. What are the ""Previous Message Ignored"" logs? Why is it getting messages and ignoring them? Is the sensor sending out multiple updates and that is clogging up the network?

[Sirmeili]

A final thought, and I don't really think this would have any effect. HS is currently running on a machine with an i7-6700k while the Polisy and HA/Hub are on much lower power machines. I woudln't think that this type of processing would be that greatly effected by this (that processor is overkill for this and that server is/was used for other things like PLex), but throwing it out there.

[Javi]

Did you actually test this on HS in the current environment?  

[Sirmeili]

25 minutes ago, Javi said:

Did you actually test this on HS in the current environment?  

I did not move all my devices back over, no. I tested with the 3 devices I'm moving around for this test. That said, HS still has about 10 other devices on it and the ISY has  a lot more. But when using the ISY event viewer, I can tell you that I don't have a lot of other traffic when I'm running my tests.  

I did the same test with all 3 devices in all 3 systems and had the same results in the ISY and HA, but not HS.

[bpwwer]

re: Previous Message Ignored

The Insteon protocol can route messages between devices.  

Device A sends a message to Device F but there might not be direct communication between A and F.  So when device B sees the message it repeats it.  Then device D sees the message and repeats it again and finally device F sees the message.

The PLM attached to the Polisy, is just another device so it may see that same message repeated multiple times.  It keeps track of the messages it sees and will ignore duplicate messages.

This is also why you may be seeing delays.  If device A is your PLM connected to the Polisy and it takes multiple repeats for the message to get to the final device.  The Polisy doesn't respond back to the REST call until the device has responded back. So the message has to propagate from the PLM to the device and back. 

You'd have to ask the author of the HomeSeer Insteon plug-in to be sure. But it's very possible that what you had configured there was an Insteon scene or the plug-in doesn't wait for a response from the device before allowing the next command.

With the thread getting as long as it it, I'm not sure if you provided the details or not, but it sounds like you have something like the following:

switchlinc light switch in room with multiple lights (switchinc's or lamplinc, etc.)

You're trying to set things up in HA so that:

You use the double tap (fast off) of the switchlinc as a trigger condition such that:

if switchlinc fast off then
  send off to lamp 1
  send off to lamp 2
  send off to lamp 3

When you double tap, the lights go off in a sequence vs. all just going off at once.

If so, there are two ways this can be accomplished with Insteon:

1) Using external logic, when the trigger condition is seen, send direct commands to each device to turn them off (like what you're doing with HA).  This is likely always going to have delays. Some Insteon controllers may have more delays than others, depends on if the controller waits for the status update from each device before allowing the next command to be sent or not.

2) Use an Insteon scene (or group). This gets programmed into either the controller device or the PLM such that when the trigger event occurs the PLM sends a single broadcast command to all the grouped devices.  All the devices see the command that roughly the same time and all respond to the command at roughly the same time.

It's very possible that on HomeSeer you were using option 2 and now you're using option 1.    It's possible to configure option 2 in the IoX.   (Or in HA if HA is controlling the PLM (or an Insteon Hub) directly).   

[Sirmeili]

6 minutes ago, bpwwer said:

 You're trying to set things up in HA so that:

You use the double tap (fast off) of the switchlinc as a trigger condition such that:

if switchlinc fast off then
  send off to lamp 1
  send off to lamp 2
  send off to lamp 3

When you double tap, the lights go off in a sequence vs. all just going off at once.

If so, there are two ways this can be accomplished with Insteon:

1) Using external logic, when the trigger condition is seen, send direct commands to each device to turn them off (like what you're doing with HA).  This is likely always going to have delays. Some Insteon controllers may have more delays than others, depends on if the controller waits for the status update from each device before allowing the next command to be sent or not.

2) Use an Insteon scene (or group). This gets programmed into either the controller device or the PLM such that when the trigger event occurs the PLM sends a single broadcast command to all the grouped devices.  All the devices see the command that roughly the same time and all respond to the command at roughly the same time.

It's very possible that on HomeSeer you were using option 2 and now you're using option 1.    It's possible to configure option 2 in the IoX.   (Or in HA if HA is controlling the PLM (or an Insteon Hub) directly).   

Actually I have a scene setup in the ISY that only has responders to turn them off (so they all turn off at once). From HA I trigger the scene Off via the ISY when I get a fast off for a specific KPL.

I have a feeling that based on my testing with the door sensor, it would be the same if I had a program in the ISY do it.

In both systems I was using Option 2, but the problem is that the scene command is not sent out, much like the individual command, until about 1.5s after the initial "trigger"  when using the ISY (whether it be the door sensor status or the fast off command from the KPL).

In HS this was faster, perhaps for the reasons you mentioned. Seems to just be a limitation of the technology. 

The reason I noticed it with the door sensor was that in my utility room I used to have a z-wave switch, but the door sensor was a hidden insteon door sensor. Opening the door, even though it had to go through HS was near instant*. When I went to both insteon devices (door and switch), even though yes, I still had the isy control the devices instead of a scene, it was a lot slower.

I understand the potential explanations why that may be given the nature if Insteon.

*I know this is not the same situation as this thread discusses. I'm just saying how I had it set up before for that door. For my main use case of using fast off from a KPL to execute an insteon scene from HS, it is still significantly faster than it is in the ISY/HA

[Sirmeili]

I find it odd that there has been a lot of talk about isy making sure that a device responds first. I have another post from last week where it seemed that the air gapping of the KPL fixed it, but the problem came back today. I send a scene to adjust the KPL buttons state and in the ISY they say they are correct, but the KPL itself never reports the same state.

I would expect from what I'm hearing here that the ISY would wait for confirmation that the KPL made it's change before updating itself.

So maybe my network is hosed. Not sure. This was all just working under HS and now under the ISY it's not. And i'm not trying to say HS is better, I don't feel it is.

I didn't factory reset any devices, but I did tell the ISY to wipe any existing links when adding these devices.

[bpwwer]

Ahh, I think I now understand better. I was not understanding where the delay was happening before and I think I do now.

For the simple case of:  if trigger then scene command it should be similar for all systems.  There could be differences in the time between the actual trigger and the system detecting the trigger, but once the system (HS, HA, or IoX) detects the trigger there should not be any delay between sending the scene command.

Maybe we've just looked at too many different things here and I'm  now confused

Going back to your initial post, it looks like that does not involve any Insteon scenes, just two devices.  This is what I see there:

device 32 86 20 sends a DON 1.  When HA detects that it calls IoX to send a DON to device 50 D1 7F.

From the pyISY timestamps, there's a delay between when the command is sent and the response is received.  Question: This delay is also visible in 50 D1 7F's behavior?  I.E. you see the send logged and 1.5 seconds later you see the light change state when the response is received?

If this is an accurate description of the issue, then I'm not sure we can debug farther as the IoX doesn't seem to log with enough precision.  We don't know when in the 1.5 second window, the IoX is sending the command to the PLM.

It could be sending the command immediately and is then waiting 1.5 seconds for a response from the device.  Or it could be waiting for incoming messages from the PLM to be processed before sending the command to device.

I don't remember if you said you can reproduce this with an IoX program in place of the HA automation or not.  If you can, document the devices and open a ticket.  That should be enough for someone to reproduce this in a IoX debug environment.