Home Assistant - Dealing With Insteon Scenes

[Venicenerd]

I'm using the ISY 994 and the the Home Assistant integration to bring all of my Insteon devices into Home Assistant. From there I bridge them over to HomeKit so I can control my lights via Siri.  I'm struggling getting scenes to work properly. I've tried two ways:

 

I. HOME ASSISTANT/HOMEKIT SCENES

I create scenes directly in Home Assistant and HomeKit and place all of the individual Insteon devices inside the scene. Unfortunately, this does not seem to work reliably for me. As soon as there are more than 3-4 Insteon devices in one scene some of them don't fire when I execute the scene. I have heard some people say that this is due to Insteon network congestion. I wanted to see if everyone else here has the same experience or if perhaps there is something wrong with my setup? I'm not trying to create massive scenes. Up to 10 Insteon devices would work for most of my scenes.

 

II. ISY/INSTEON SCENES

Here I create a scene on the ISY itself and then bring that scene into Home Assistant as a switch:

Then I bridge those switches over to HomeKit. So now when I say "Hey Siri, set Sunset lights" the "HA Scene Sunset Lights" switch in Home Assistant turns on. The ISY then turns off the six lights specified in the scene, and turns on the "Bedroom Deck Door" and "Front Door" at 100% brightness. So far so good! 

The problem arises when I change just one of the lights manually. If, for example, I turn off the "Kitchen Deck" light (either by switch, inside ISY, HA, or HomeKit) the "HA Scene Sunset Lights" switch inside HASS will remain in the "on" position. Therefore, I won't be able to execute my scene via HomeKit again since both HomeKit and Home Assistant believe the scene is still active. The switch is already "on" so obviously they will not try to turn it on again.

This can get really messy as certain lights are part of multiple scenes. So as soon as things become out of sync one can no longer, reliably execute scenes this way. Also if you have two scenes with the same Insteon devices but but each set to different brightness levels this will not work. 

Is there no way to specify that Home Assistant should only consider the "ISY Scene Switch" as turned on when the scene is in the EXACT state that you have specified on the ISY? If any of the lights are not in the EXACT state (on/off, brightness, etc...) it should be displayed as "off".

 

SUMMARY

Obviously I would prefer to just create all of my scenes natively inside Home Assistant and HomeKit itself. Is this truly not possible with ore than 3-4 Insteon devices at a time?

If going the ISY scene is the only way I would love to find out if there is a better way to go about this so that Home Assistant only report the scene as "on" when all devices are in the exact state as specified. 

There must be a better way to deal with this, right? I am open to all thoughts and suggestion. I am sure others must be struggling with this as well?

[upstatemike]

I remember when the ISY-26 was first being developed there were long discussions about scenes and whether changing the state of one device in the scene should change the entire scene to false (off). It sounds like the Home Assistant folks have not really hit on a strategy yet for addressing the situation you are describing. Maybe the scene object in Home Assistant needs to send a scene command and then immediately return to a neutral state so that you can either turn the scene off or send another on to restore the scene from any manual intervention.

[xlurkr]

And it is complicated, because with its cleanup messages, Insteon takes the approach that it will keep trying to get the correct state of all devices in a scene for a while.  I'm not sure how many other protocols do that.  The bigger issue is what you do when one device, temporarily or permanently, doesn't achieve the state dictated by the scene's definition.

I personally don't see any downside to always having the controller attempt to set the state of every device in the scene (whether by individual or group messages) every time it is invoked regardless of what it thinks the state currently is, excluding super-complicated environments where minimizing traffic is essential.  Otherwise, correcting this "scene out of whack" scenario requires one to invoke an unwanted scene just to be able to invoke the desired one.  But if that behavior isn't built into the highest-level controller you interact with (like Siri invoking Homekit), I don't know what you can do about it other than interact directly with the software layer that does allow for it.  The ISY will do what it's told to do.  It doesn't ignore your commands because it thinks they're redundant.

However, I've also never seen the value of totally accurate scene awareness, either.  Though it's helpful to glance at a display of scenes to know the state of devices that are not in line-of-sight or aren't lights so they don't provide their own visual feedback, it seems more like a "nice to have" than something essential.  But then I live alone, so no one is around to manually change a light and screw up my desired lighting, or themostat setting, etc.

-Tom

[MrBill]

5 hours ago, Venicenerd said:

I have heard some people say that this is due to Insteon network congestion.

Absolutely true!  When an ISY scene is turned on or off, there is a single transmission sent on the Insteon network.. "Turn on Outside lights"... the switches that are part of the scene are in fact listening for both commands that are part of the scene and commands being sent to that switch itself.  This is a feature.  It not only reduces communication congestion but it also allows the system to work when the controller is offline. 

Also as far as the ISY is concerned scene's are also stateless.  The ISY doesn't know if a "scene" is on or off, it simply sends a scene on or off command when told to.  Over the years various mobile devices (as well as home assistant) have tired to derive a state for scenes, some of them had better idea's than others and some don't really work too well.  

On the contrary, when HA controls an HA scene there is alot of traffic both between ISY and HA and again over the insteon network, because HA needs to tell each light what to do individually.

With home assistant I haven't had to do this in most cases, but there are a few where I need ISY helper programs to control the scene's from HA.   in the simplist form create an ISY program for the scene:

Program UniqueProgramName

If
    (nothing)
then
    Turn On Outdoor Scene
else
    Turn Off Outdoor Scene

Save the program on the ISY side, then either Reload ISY or Restart Core on the HA side.

Now create an On Button and Off Button

show_name: true
show_icon: false
type: button
tap_action:
  action: call-service
  service: isy994.send_program_command
  data:
    name: uniqueprogramname
    command: run_then
  target: {}
name: Outside On

and

show_name: true
show_icon: false
type: button
tap_action:
  action: call-service
  service: isy994.send_program_command
  data:
    name: uniqueprogramname
    command: run_else
  target: {}
name: Outside Off

You might couple these buttons in the Dashboard with the new Mushroom Chip cards which could be used as a status indicator for each member of the scene.

An important note:  When naming programs on the ISY side the name must be unique, the folder structure is not scene on the HA side.  So for example if a program named "Bill Lamp" existed in ISY program folder structure in both the Bedroom and the Great Room, HA can't distiguish those programs names instead each program name must be unique "Greatroom Bill Lamp" and "Bedroom Bill Lamp".   Note also Case counts..."Bedroom Bill lamp" and "Bedroom Bill Lamp" are not the same.

 

 

Edited October 19, 2022 by MrBill

[upstatemike]

What does HA do with ISY names in the form room-character-device such as Bedroom:Table Lamp vs Den:Table Lamp ?

[MrBill]

54 minutes ago, upstatemike said:

What does HA do with ISY names in the form room-character-device such as Bedroom:Table Lamp vs Den:Table Lamp ?

Should work fine translating node names to entities, it may change the : to an underscore (I don't use : in node names, but my guess it becomes an _ ).  With nodes HA will translate "folder" to a "location", if the entity name already exists the second will get a _1 or _2 at the end or something at the end. 

However with ISY programs there is a single name space, and folders are ignored as part of the naming.  If there's a duplicate then Send Program Command will find the first in it's namespace and ignore duplicates.

[shbatm]

Quote

I create scenes directly in Home Assistant and HomeKit and place all of the individual Insteon devices inside the scene.

Echoing what others have said - Insteon Scenes (and actually Zigbee Groups and Z-Wave Associations) can be stored natively in the devices, so it only takes 1 network command to trigger a scene/group. Trying to create the scene/group in Home Assistant just results in individual calls to all of the devices which causes significant IP and insteon network traffic. Wherever possible, you should try and use the native groups.

13 hours ago, Venicenerd said:

If going the ISY scene is the only way I would love to find out if there is a better way to go about this so that Home Assistant only report the scene as "on" when all devices are in the exact state as specified. 

To handle this, Home Assistant exposes a state attribute "group_all_on" for each group/scene -- this is only "on" if all of the devices in the group are ON--not a guarantee that they are matched with the scene, but at least handles the case where one device of many is on and the group is shown as "on".

More details here: https://community.home-assistant.io/t/isy994-hacs-version-with-pyisy-beta/166897/4?u=shbatm

For scenes that should really be scenes (e.g. set lighting levels, not just a group of devices) in Home Assistant Dashboards I've found that its easier to use buttons and assign service calls (switch.turn_on / turn_off), rather than rely on the toggles from an entities card. Sending a "turn_on" service call to an ISY Scene will still cause it to trigger the scene--even if it already shows as "on" in Home Assistant.

Also as an aside for keeping states in-sync when manually adjusting--adding in a link to an older post on "follower programs" that I use. The short version is--I use ISY programs to "watch" devices that should be in-sync, if they don't match after 2 seconds (which they should if the command was inside the insteon network/direct link, but not if it was a Home Assistant/Alexa command to a single device) the program updates the other statuses to match.

Edited October 19, 2022 by shbatm

[steve053136]

I have all my scenes created in home assistant and bridge them over to google home. This was working fine for the past couple years until I upgraded homeassitant from 2022.6 to 2022.8

I could always tell there was some congestion with the link between ISSY and homeassistant. For example: I would call 10 lights on and they would turn on 1 by 1 with maybe a second or less between each one but they would always all come on. I have 40 insteon Din rail dimmers so turning all lights off at night for example would take a little bit (~20 seconds) I had no issues with the lag. I never had any Sync issues running like this. 

Now after the update of homeassitant version, after like 3 or 4 lights come on it gives up stops. This has broke many of my automations and I no longer can use google home to control more than 3 lights at a time. Even calling more than 3 or 4 lights from the UI no longer works 

 

[MrBill]

10 minutes ago, steve053136 said:

I have all my scenes created in home assistant and bridge them over to google home. This was working fine for the past couple years until I upgraded homeassitant from 2022.6 to 2022.8

I could always tell there was some congestion with the link between ISSY and homeassistant. For example: I would call 10 lights on and they would turn on 1 by 1 with maybe a second or less between each one but they would always all come on. I have 40 insteon Din rail dimmers so turning all lights off at night for example would take a little bit (~20 seconds) I had no issues with the lag. I never had any Sync issues running like this. 

Now after the update of homeassitant version, after like 3 or 4 lights come on it gives up stops. This has broke many of my automations and I no longer can use google home to control more than 3 lights at a time. Even calling more than 3 or 4 lights from the UI no longer works 

 

Give this thread another read, your answer is there.  Use insteon/ISY scenes and control them via home assistant.

[vbPhil]

 

On 10/19/2022 at 4:56 PM, shbatm said:

Echoing what others have said - Insteon Scenes (and actually Zigbee Groups and Z-Wave Associations) can be stored natively in the devices,

I'm not understanding how to store a link natively in a device. I must have missed that in my upbringing.

For IoX scenes that I don't need in HA I'm disabling the entity for that. In doing so I've realized I have many scenes set up to control one responder (a lamp dimmer module) and a couple mini key panel switches as controllers. Perhaps that's not a really good use of a scene, having only one responder, and I could use a program instead maybe making HA more compatible.

In your opinion does that sound reasonable or am I approaching this incorrectly? 

[shbatm]

48 minutes ago, vbphil said:

I'm not understanding how to store a link natively in a device. I must have missed that in my upbringing.

When you create a scene in the ISY Admin Console, it "writes" the scene to the devices. Those are the native links. The Insteon device then listens for both commands to "Turn myself on" and "Turn on if Scene 1 is told to turn on"

So in short: if you make a group in Home Assistant of [Light 1, Light 2, and Light 3] and try to turn them on, it sends 3 commands--one to each light. If you make the same Scene in the ISY Console (which makes the native links) and turn on the scene "switch" from Home Assistant, it will only need to send 1 command since Light 1, 2, and 3 are all listening for that scene.

48 minutes ago, vbphil said:

For IoX scenes that I don't need in HA I'm disabling the entity for that. In doing so I've realized I have many scenes set up to control one responder (a lamp dimmer module) and a couple mini key panel switches as controllers. Perhaps that's not a really good use of a scene, having only one responder, and I could use a program instead maybe making HA more compatible.

Having extra scenes is fine. You can either disable them in Home Assistant, make them Hidden (so they don't show up on dashboards, but can be used in automations), or use the "Ignore String" setting to prevent Home Assistant from loading them in the first place.

I can't find my other post, but in general--I "use" the load devices in Home Assistant/Alexa for control by a user, and then have follower programs "fix" the status of things like other KPL buttons or 3-way slaves.

If I have to control more than 2 load devices at any time, I use an ISY scene.

Here's my cheatsheet that I use:

Exposed Devices to 3rd Party Controllers

• Expose the load-controlling device for dimmable Insteon Devices
• Expose the Scene for On/Off devices

 

Adding to ISY

• Add the load-controlled device and all controllers to a scene
• Add a state variable for tracking the brightness "s.Dim.<<DEVICE>>"
• If devices do not have a keypadlinc secondary button:
  • Add 2 programs
    • Device Brightness
    • 
    • Device Slave
    • 
• If devices have a keypadlinc secondary button to track
  • Add the controllers to a "Fix Scene" scene that excludes the load-controlled device
  • Add 2 programs
    • Device Off
    • 
    • Device Not Off
    • 

Edited February 1, 2023 by shbatm

[vbPhil]

13 minutes ago, shbatm said:

When you create a scene in the ISY Admin Console, it "writes" the scene to the devices. Those are the native links.

Okay, I had that understanding. I thought maybe you were referring to another way to link a responder and controller without creating a scene. It's the IoX scene node that HA is picking up and not behaving as it does in IoX. Thanks for the other info. Seems like allot of overhead to add in IoX.

 

20 minutes ago, shbatm said:

make them Hidden (so they don't show up on dashboards

One thing I found with hiding an HA entity is that it may still shows up in the custom:auto-entities card but a disable worked to keep it off the card.

[shbatm]

23 minutes ago, vbphil said:

One thing I found with hiding an HA entity is that it may still shows up in the custom:auto-entities card but a disable worked to keep it off the card.

exclude:
- hidden_by: user

 

[vbPhil]

8 minutes ago, shbatm said:

exclude:
- hidden_by: user

I couldn't get that to work with this configuration. The first exclude worked but not the hidden by.

Quote

type: custom:auto-entities
card:
  type: entities
  title: Great Room
filter:
  include:
    - domain: light
      area: Great Room
      options:
        type: custom:mushroom-light-card
        layout: vertical
        icon_type: none
        collapsible_controls: true
        fill_container: true
        show_brightness_control: true
        use_light_color: true
        show_color_temp_control: false
        show_color_control: true
        entity: this.entity_id
  exclude:
    - device_manufacturer: Signify Netherlands B.V.
    - hidden_by: user
sort:
  method: device
  reverse: true
  ignore_case: true

 

[vbPhil]

1 hour ago, vbphil said:

Perhaps that's not a really good use of a scene, having only one responder, and I could use a program instead maybe making HA more compatible.

I must answer my own statement here. That is a good use for a scene because it links the responder and controller in the devices so that the link works even without the IoX controller.

[vbPhil]

So its been a few months and during that time I've added a Insteon USB PLM to HA for testing how that behaves. I have a question on scenes. When I create an Insteon Scene in HA the links are not written to the devices like they are with the ISY. To clarify this, I don't mean an HA Scene but rather the Insteon Scene created in HA through the Insteon Panel. Is there another step I'm missing or does HA not write the links to the devices?

[brians]

On 4/17/2023 at 9:15 AM, vbPhil said:

So its been a few months and during that time I've added a Insteon USB PLM to HA for testing how that behaves. I have a question on scenes. When I create an Insteon Scene in HA the links are not written to the devices like they are with the ISY. To clarify this, I don't mean an HA Scene but rather the Insteon Scene created in HA through the Insteon Panel. Is there another step I'm missing or does HA not write the links to the devices?

I tried a RF only Insteon USB stick and tried the HA Insteon Integration. Can't manage devices... seems no way to remove once they are added. It looked promising at first, I added my keypadlinc just by entering the device ID, it added and I was able to control. But nowhere in there is a way to manage devices. I ended up removing the integration, which fortunately removed the device and entities. I will continue to use Polisy and the HA ISY integration since it works so well.

 

[blueman2]

On 4/17/2023 at 9:15 AM, vbPhil said:

So its been a few months and during that time I've added a Insteon USB PLM to HA for testing how that behaves. I have a question on scenes. When I create an Insteon Scene in HA the links are not written to the devices like they are with the ISY. To clarify this, I don't mean an HA Scene but rather the Insteon Scene created in HA through the Insteon Panel. Is there another step I'm missing or does HA not write the links to the devices?

@vbPhil, glad you posted this.  I am seeing the exact same thing.  HA does not create a scene the same way as ISY does.  It appears the ISY creates native Insteon links between all the devices in a scene, so that even if the PLM is removed, the devices still talk to each other.  HA on the other hand just creates a link between the devices in the scene to the PLM, and gives it a group # for use in HA.  So if the PLM is disconnected for any reason, the scenes no longer work.  I like the way ISY does it much more, since I want my house to still work when the PLM is dead for some reason.  

 

Maybe we can ask the Insteon plug-in leader in HA to add the creation of native links in the Insteon devices when HA creates scenes.  

[brians]

15 hours ago, blueman2 said:

@vbPhil, glad you posted this.  I am seeing the exact same thing.  HA does not create a scene the same way as ISY does.  It appears the ISY creates native Insteon links between all the devices in a scene, so that even if the PLM is removed, the devices still talk to each other.  HA on the other hand just creates a link between the devices in the scene to the PLM, and gives it a group # for use in HA.  So if the PLM is disconnected for any reason, the scenes no longer work.  I like the way ISY does it much more, since I want my house to still work when the PLM is dead for some reason.  

 

Maybe we can ask the Insteon plug-in leader in HA to add the creation of native links in the Insteon devices when HA creates scenes.  

I already put request about my issue not able to delete devices but no response. Insteon doesn’t seem every popular anywhere else (not really popular anywhere now actually) therefore not much incentive or interest to support probably. I stick with Polisy and ISY integration and my plan is to switch out Insteon 37 remaining devices little by little as required. Insteon scenes for me are not necessary and I only used for 3 ways and switchlincs, but I am replacing those with better things. 

[shbatm]

18 hours ago, blueman2 said:

I am seeing the exact same thing.  HA does not create a scene the same way as ISY does.  It appears the ISY creates native Insteon links between all the devices in a scene, so that even if the PLM is removed, the devices still talk to each other.  HA on the other hand just creates a link between the devices in the scene to the PLM, and gives it a group # for use in HA. 

Just an FYI -- this is a limitation of Home Assistant, not just the Insteon or ISY integrations. There's been several discussions over the last years about how to enable "native" groups/scenes (Insteon, ISY, Zigbee, Thread, Z-Wave all have some variation), but the choice has been to maintain Home Assistant scenes as self-contained to Home Assistant and just act as "state-recall" scenes. You can look for some of the discussions on https://github.com/home-assistant/architecture.

[blueman2]

Thanks, @shbatm.  And thanks for all the work you do and the leadership you have shown in this area.

It does make things easier in a way to not store "native" scene info on the devices.  Ensuring that the native records and the local HA records stay in sync can be an issue.  And I can always edit the ALDB or using the UI or doing service calls in HA to write the links directly to the devices.    The issue really comes up though where I have 3-way switches and I want the switches to control each other.  Or where I have an "all outside devices" scene that I want controller by a switch.  

[ccclapp]

Hi

This is a great discussion on a complicated topic without a great resolution.  Assuming the info in this thread is still valid I created a new Discussion summarizing this in the HA Architecture github, here..

https://github.com/home-assistant/architecture/discussions/946

Since that is not really a general discussion forum, I recommend we wait to see  how HA developers etc reply.  Im posting here so you can also be informed if they do.

As to context: I've been a long-term ISY-only user.  I have many many lights (switches) with many scenes. I've just completed a significant home renovation, so have had isy offline for about a year and am now researching what direction to go in moving forward.  It seems for lighting Insteon/ISY may be best. But for the overall front-end and much broader device control HA will likely be my primary front end.  Ive similarly been trying to evaluate Insteon vs Z-wave vs Zigbee for lighting and other purposes (energy monitoring, sensors etc) and may stay Insteon for ligating and different for the rest.  Here again we get to the question of what native controller/hub for each protocol: ISY, HA, Other and then which to link through which....BUT all of this will likely be a separate thread (as its somewhat off-topic here), here and at HA...

...however, one quick more related question: Am I correct there is decent ISY integration in HA, but not an equally good integration to being all HA devices into ISY/eisy if one is considering isy as the primary front end?

Thanks again for the great info in the discussion above. I wish there was a good and simple answer

Edited July 17, 2023 by ccclapp

[ccclapp]

Im adding a 2nd post here on topic:

1. As stated in this thread, isy/IoX writes scene and other data natively to Insteon devices. Also stated, that Zigbee Groups and Z-Wave Associations can be similarly natively written to devices.  My question is: Does ISY/eisy do this with those protocols, or only with Insteon.

2. And: if written natively (by ISY or otherwise) does that help even the playing field between insteon/z-waye/zigbee for lighting (scenes etc)? (not meaning to open a big discussion about pros/cons of each here)

Thanks

[ccclapp]

Sorry for another follow-up post..

My post to the HA Github discussion was closed by ../Frenck without addressing the question

I therefore made a similar post in the general forum here https://community.home-assistant.io/t/scenes-groups-associations-store-natively-in-device/592910?u=ccclapp if anyone cares

[Geddy]

@ccclapp unsure what you want from here either. But this thread was basically dead. It would be best to start a different discussion here. You're taking onto a topic that started almost a year ago and had stalled out a couple of times.