How to test Insteon devices for ALL ON vulnerability?

[kzboray]

Can someone suggest a way to test Insteon devices for the older firmware that supported ALL ON events? As a bonus if I could do this from a console and not have to physically remove devices to see the Rev. version that would be a huge win.

[Techman]

The device's firmware shows up in the device tree if you click on Network or ISY.

I'm not sure what versions of a specific device's firmware removed the "all on"

[kzboray]

@Techman Yes, thank you. AC shows the firmware version, but not the revision code. And as you so aptly pointed out, knowing what versions of firmware actually have this vulnerability is the next step. @IndyMike started to document this by testing a variety of devices. I figured I'd have to do the same with the devices not on his list. I need to test approximately 129 devices. I'd like to cut that list down to a more manageable size by first identifying which versions of FW still have the ALL ON enabled but short of that I'll need to start testing myself.

To do that I need a tool that will allow me to send a special packet (unknown as of yet) to trigger the ALL ON firmware code. IndyMike mentions something about sending FF, but I suspect there's more to it than that.

So I started this tread hoping a few of my fellow propeller heads had some ideas.

[Techman]

@kzboray

Send an email to Steve Lee at Insteon:  slee@insteon.com

He has been with Insteon for a long time and probably has the answers to your questions.

I believe that the "all on" was an Insteon firmware issue not a hardware issue. If that's the case then it should make your project a lot easier to debug.

[kzboray]

@Techman Thank you mate. I'll do that!

 

[paulbates]

DIdn't versions of ISY AC/FW maybe 10 years ago have an ALL ON button? If yes and you have a "dev" ISY you could load old firmware to do your testing. 

I don't remember exactly, but maybe a version in this timeframs

 

[IndyMike]

I am currently using the ISY994 with firmware V5.3.4.  It contains All-On/All-Off buttons on the "My Lighting" and "My Network" Tabs.  Press either of these and you will get a Broadcast Group All-On/All-Off command to group "FF".  For Insteon, group FF is ALL DEVICES.

I checked again this AM.  The following works like a charm.  Hit All-On or All-Off and all of my older devices respond.  My newer devices (V.45) do not.  The short transmission in the event viewer is the Broadcast group command used (I used All Off)  I highlighted the Group # (FF) in the command to show it was for all devices.

This is not an error or deficiency.  The broadcast group command is part of documented Insteon protocol and group xFF is defined as all devices.  What is an anomaly is the "collision" or other event that causes the PLM to transmit the command when something else is being requested (I am assuming here).

There are other serial port tools that will allow you to communicated directly with the PLM to generate to command shown below.  I actually wrote one using VB years back.  The good news is that you don't need them.  The ISY994 can do this from the Admin Console.  I'm hoping the PolyIsy and Eisy can also. 

 

 

[IndyMike]

Snippet from the Insteon developers guide defining All-Link Groups (Group broadcast).  Group broadcast is used Every time you activate a Scene.  The All-on/All-off command is simply a Scene with a Group #FF (all linked devices).

Edited January 12 by IndyMike
Highlighting

[vbPhil]

I'm confused about the phantom All On phenomenon. I just had one this morning where all my Insteon devices came on, at least I think all of them. I was still in bed asleep, and it woke us up. In a sleepy stupor I just started turning everything off from my iPhone without troubleshooting.

Is the phantom All On only turning on those older devices that have the All On function or will all devices new and old respond to that All On command?

[IndyMike]

@vbPhil,

Old devices will activate.  We don't have great answers on newer devices.  So far I3 devices appear to be immune. 

I tested a number of Dimmers/relays/KPL's and devices with firmware v.45 were immune. 

You can test your system by using the "all-on/all-off" button in the My Lighting admin screen.  It's best to disable programs prior to the test. 

Important:

Also, the ISY will ASSUME that all devices turned on/off since it issued the command (you'll see all device on in the summary screen).  You  need to follow with a system query to see which devices were affected.

[vbPhil]

18 minutes ago, IndyMike said:

You can test your system by using the "all-on/all-off" button in the My Lighting admin screen.  It's best to disable programs prior to the test.

Yup, I've got an awful lot of pre v.45 firmware and they all came on with the  All-On button.

Also, my I3 Paddle v.54 and a 2635-222 On/Off Module v.48 came on.

[IndyMike]

3 minutes ago, vbPhil said:

Yup, I've got an awful lot of pre v.45 firmware and they all came on with the  All-On button.

Also, my I3 Paddle v.54 and a 2635-222 On/Off Module v.48 came on.

That's a rather large oh excrement...

The I3 paddle and the On/Off module reacting are news items.  If you don't mind a few additional questions -

1)  Did you visually verify that the devices activated?

2) If not visually, did you perform a follow-up query on the devices to make sure they actually responded?

Thanks - it's been many years since this first raised it's head.  We're still learning,

IM

[vbPhil]

12 minutes ago, IndyMike said:

1) Did you visually verify that the devices activated?

2) If not visually, did you perform a follow-up query on the devices to make sure they actually responded?

The I3 Paddle did not actually come on but the On/Off module did.

The follow up query shows the I3 Paddle Off and the On/Off Module On.

[IndyMike]

1 minute ago, vbPhil said:

The I3 Paddle did not actually come on but the On/Off module did.

The follow up query shows the I3 Paddle Off and the On/Off Module On.

Good news on the I3 device.  So far they are clean.

Disconcerting that a 2635-222 On/Off Module with firmware v.48 came on.  I'm thinking that device was produced well after my V.45 dimmers with a date code of 1616.  It means we don't know much about incorporation.

Wondering if there is a database expert out there who would volunteer to take on the job of tabulating susceptible devices.  I'm a simple ignorant Engineer, so I'm not even sure what I am asking.

Here's my data to start - https://forum.universal-devices.com/topic/41651-all-on-removed-in-what-firmware-version-of-switchlinc-dimmers/?do=findComment&comment=369748

[vbPhil]

Seems like we need new firmware in the PLM to not issue that phantom All On. Is Insteon aware of this issue? I'm probably late to the party and all this has already been said.

[IndyMike]

26 minutes ago, vbPhil said:

Seems like we need new firmware in the PLM to not issue that phantom All On. Is Insteon aware of this issue? I'm probably late to the party and all this has already been said.

Agreed - I had thought that this was implemented years ago at the PLM level.  I'm rather slow to update hardware.

I had previously implemented mitigation as recommended by: https://wiki.universal-devices.com/INSTEON_Random_All_On_Events

The procedures outlined did work for me.  That was many years ago. 

Things started popping up again about one year ago.  I ran a few tests, and was astounded to find that my "new" PLM still had the command in firmware, as did most of my Insteon devices.  I have no idea whether newer "hubs" have the command removed. 

Edited January 26 by IndyMike

[paulbates]

13 minutes ago, IndyMike said:

I ran a few tests, and was astounded to find that my "new" PLM still had the command in firmware, as did most of my Insteon devices. 

Mine says "Rev 2.7 2023". I bought it in October 23 and as we discussed the other day, pressing the All On button in iox had an effect on more devices than it should. Knocking on wood as I say: "I no longer have phantom ons or offs". I do have the potential for them

[vbPhil]

My PLM is a new USB unit bought on 4/17/2023 from Insteon with firmware 9E running with Polisy Pro. I've had about only 3 phantom All On incidents, all this year, and I've been using insteon >20 years. I can't say for sure but they may have only happened since the new USB PLM.

[IndyMike]

6 minutes ago, paulbates said:

Mine says "Rev 2.7 2023". I bought it in October 23 and as we discussed the other day, pressing the All On button in iox had an effect on more devices than it should. Knocking on wood as I say: "I no longer have phantom ons or offs". I do have the potential for them

I believe @Brian Hindicated that the Rev 2.7 PLM's used the same firmware as my older Rev 2.1 PLM (V.9E date code 1514).  It's been around a long time. 

[ELA]

2 hours ago, vbPhil said:

Seems like we need new firmware in the PLM to not issue that phantom All On. Is Insteon aware of this issue? I'm probably late to the party and all this has already been said.

My best understanding of the All On phenomenon is that it is caused by data collisions at the signal level.   I have not heard of a PLM issuing a phantom All On command itself.

     It was my understanding that Insteon was removing the ability of devices to react to the All On ( Group broadcast to group FF) over time.     Until all devices firmware's were updated the All On phenomenon could remain an issue.

What I have not heard about is:  whether or not the ability of a device to repeat a Group Broadcast message to group FF was also being removed?   One would hope so.

 

[IndyMike]

1 hour ago, ELA said:

My best understanding of the All On phenomenon is that it is caused by data collisions at the signal level.   I have not heard of a PLM issuing a phantom All On command itself.

     It was my understanding that Insteon was removing the ability of devices to react to the All On ( Group broadcast to group FF) over time.     Until all devices firmware's were updated the All On phenomenon could remain an issue.

What I have not heard about is:  whether or not the ability of a device to repeat a Group Broadcast message to group FF was also being removed?   One would hope so.

 

@ELA,

I had originally thought that the data collisions occurred on the powerline as well.  We always questioned whether the CRC shown in the Insteon protocol actually existed.  In recent years I have re-read some of LeeG's posts and come to the realization that collisions at the PLM/ISY interface are far more likely.  This would explain how corruption could occur with a correct CRC - the PLM ADDS the CRC after the corruption.

When the ISY issues a broadcast group command to the PLM it takes the form (fast on):

02 62 00 00 XX CF 14 00  (Where XX is the group number to activate)

The PLM transmits the following onto the power line (per the Insteon Message Summary above)

YY YY YY 00 00 XX CF 14 00 ZZ  (Where YY YY YY is the PLM address and ZZ is a CRC).

For a random collision on the powerline to produce YY XX and ZZ would be extremely unlikely.

If we assume a collision at the PLM powerline (assume the plm address YY YY YY is intact) we still need a valid XX and a correct ZZ (crc).  Still rather unlikely.

Is seems far more probable that the collision (at the plm/isy interface) affects the XX group number.  Assuming it was a valid group number for the PLM, it would simply append the PLM address (YY) calculate the crc (ZZ) and put it on the powerline.

That's where my head is today.  But then I've rethought this many times over the past 10 years.  I probably need a new hobby.

I had not thought about preventing devices from repeating FF (all device) transmissions.  Since all devices repeat transmissions regardless of whether they are addressed, I am not sure this is possible.  Absolutely worth asking the question. 

 

[ELA]

Thanks IndyMike,

I had not heard about the RS232 interface as the possible culprit.   Kind of a large oversight if that is the case since RS232 handshaking would prevent that if they had implemented it.

[IndyMike]

@ELA,

There is no hardware handshaking (RTS/CTS, dRTS/dCTS) in the PLM serial interface: https://cache.insteon.com/documentation/2413Sqs-en.pdf.

I am not sure that the problem is actually in the serial interface.  The serial protocol requires the modem to acknowledge the input command with a 06 (ack) or a 15 (nak).  This is their approach to verifying the data and preventing device overrun.   If the PLM is receiving data from the ISY when a RF transmission comes in, it should probably respond to the ISY with a NAK (or at least do nothing).

Since we have never been able to trace an actual all-on event, we are speculating.  It's possible that the data transfer between the ISY/PLM is successful, and the PLM gets a RF interrupt while it's processing the requested communication resulting in a all-on (Total WAG).

 I am not trying to start a crusade here.  Many of us have lived with these issues for over 10 years.  The issues can be mitigated to a large degree by following the suggestins on the UDI Wiki.

I am trying to warn people about the hazards of using automation devices in critical or secure applications (fireplaces, garage doors, motors). 

Edited January 27 by IndyMike

[ELA]

 IndyMike,

I am aware they did not implement RS232 handshaking.  Further the RS232 interface used is full duplex so collisions at the RS232 interface should not be possible.

    Agreed there has been a lot of speculation regarding the root cause and no need to start a crusade.

I experienced 3 All-On events, before implementing delays after IR sensor activations, a long while back.  I had one event that I am pretty sure was caused by an ESD event into the power line near an Insteon device.   Hard to explain that one.

I totally agree with your efforts to warn people about  the hazards associated with using Insteon for critical or secure applications.   I would monitor my garage door but would never use Insteon to activate- open/close it.

[larryllix]

I had a OnOffLinc module produce scene on commands randomly in response to  control commands On/Off. This plagued me for a few years until one day I recognised the pattern of brightness levels on a few lamps and investigated.

l always figured I had some bad program turning on a lighting scene (it appeared too smart). Not finding any clues agan, despite many attempts to find it in my programs, I browsed through the ISY logs this time.  I found the ACK signal from my Insteon humidifier control plug-in device logged as a bunch of noise. I searched back a few weeks and found it occurred previously at that same time.

I manually sent commands to turn the humidifier on and off several times and then checked the logs to find  an occasional lack of confirmation from the OnOffLinc and a bad packet in it's place.

A simple power cycle and factory reset of the Insteon device appeared to fix the problem and I haven't ever experienced it again, for many years now.

IIRC, 'All On' was just another pre-installed Insteon scene code  in early Insteon devices, and this was just another Scene control that was being put out by a defective device. No RS-232 involved.
 

So much for the packet security checks Insteon claimed in their white papers.

BTW: RS-232 typically uses X-On and X-Off codes to do handshaking. The modem hardware signal lines were never defined to be dynamically operated.