Everything turned ON suddenly...

[TanerH]

So, before I get into this event: a few weeks ago, I actually had the opposite happen: Everything in the house turned OFF all of a sudden... I had meant to post about it then, but neglected to.

 

This time, however, suddenly everything in the house turned ON (including my I/OLinc that triggers the door bell ).

 

Right around that time there could have been a program running to turn on 1-2 lights (evening).

 

I don't really see anything else in the exported UDReport logs around that time, other than me going around and turning things back off again. Oddly, I don't even see ALL of the switches I turned off... hmm.

 

The Error log has this near that timeframe, but I think actually AFTER the event (I honestly don't even remember when it exactly happened):

 

Fri 2013/07/12 06:41:09 PM	System	-5012	26	
Fri 2013/07/12 06:41:14 PM	System	-170001	[uDSockets] RSub:30 error:6	
Fri 2013/07/12 06:41:19 PM	System	-170001	[uDSockets] RSub:30 error:6	
Fri 2013/07/12 06:41:24 PM	System	-170001	[uDSockets] RSub:30 error:6	
Fri 2013/07/12 06:41:29 PM	System	-170001	[uDSockets] RSub:30 error:6	
Fri 2013/07/12 06:41:34 PM	System	-5012	27

 

Is there something that could trigger this, somehow? What sort of things should I look for when troubleshooting this?

 

Unfortunately, I did not have the ISY admin console open at the time, so I don't have anything from the Event Viewer :-/

 

This is the first time (other than the all-off that happened) something like this has happened since installing all my Insteon devices (about 2mo+). I did recently upgrade my ISY to 4.0.5, though.

 

Thoughts?

 

-Taner

[TanerH]

Wow, I should probably have put a few more details in that post, sorry

 

1) ISY 994i running 4.0.5

2) Roughly 60 devices, mostly SwitchLinc (switch or dimmer), 5x IOLinc, 2x KPL8, 2x OutletLinc, 1 EZIO4, 1 iMeter Solo

3) I have a SignaLinc bridging the phases near one of my sub panels

4) I have to run Access Points to properly get signal into my detached garage, as I'm guessing the power run is a bit too far (the mains power in my house is run in an "interesting" fashion, and I have solar that feeds into it, at the detached garage -- so perhaps that is why). Even that is not a stable connection, though (I often get errors that they cannot communicate).

[Michel Kohanim]

Hi Taner,

 

There has been other report of this issue and thus far the problem seems to be related to INSTEON traffic being misconstrued as group 255 (all). Question

 

Do you happen to have a program that checks the condition of a Motion Sensor and then changes the status of a KPL button (through a scene)? If so, please try putting a delay between the change in status and then the activation of the button.

 

With kind regards,

Michel

[TanerH]

First, thanks for the response!

 

There has been other report of this issue and thus far the problem seems to be related to INSTEON traffic being misconstrued as group 255 (all).

Hmm, this is traffic begin (perhaps) sent to the PLM and misconstrued as "Turn on ALL", or you mean there is malformed packets being broadcast which each device sees/acts on?

 

Do you happen to have a program that checks the condition of a Motion Sensor and then changes the status of a KPL button (through a scene)? If so, please try putting a delay between the change in status and then the activation of the button.

I don't have any Motion Sensors (the list I provided should be concise as to module types, just not exact on the count ), but I have 2 KPL8's that check states and adjust scenes based on those states.

 

A) 3 buttons check the state of various lights (kitchen, hallway, family room) and set an appropriate KPL on of off. General form of these programs is "IF (light1 status off AND light2 status off ...) THEN set KPL button scene off ELSE set KPL button scene on"

 

second KPL shows the state of my garage doors, and if a button is pressed will "press the garage door button". This is actually a slightly intricate setup:

• [*:3rdka2ta]IOLinc dry contacts connected to a magnetic switch, closed (On) when door is closed. There are 4 of these (4 doors).
  [*:3rdka2ta] 4 separate programs monitor these sensors and each sets a variable on or off based on the door state, and then calls my "StatusALL" program which sets the light on a 5th button on or off if any of the doors is open (If $door1=1 OR $door2=1 [...] THEN set scene GarageANY 100% ELSE set scene GarageANY off)
  [*:3rdka2ta]Each of the 4 status/control buttons on the KPL has an associated program that toggles its respective channel on an EZIO4 that does:
  • 
    [*:3rdka2ta]Set DoorOpen1 100%
    [*:3rdka2ta]Wait 2 sec
    [*:3rdka2ta]Set DoorOpen1 OFF

...this was because I could not program my EZIO (requires Windows XP!?!) to make the relays momentary... I am thinking to redo these to use the relay side of the IOLinc... I hadn't done that initially because this now complicates the wiring greatly.

[*:3rdka2ta]Due to the inconsistency communicating to the garage devices, I created another program, linked to a 6th button, that essentially "rescans" the sensors (calls QUERY on each sensor)

[*:3rdka2ta]Also, in case it matters, this KPL Button mode is "Non-Toggle [OFF]", since I control light state from programs

 

EZIO4O:

 

 

Relay side of EZIO4O is soldered onto the "button" contacts of the garage door controller:

 

Sorry if any of that is hard to follow -- I can give you exact code if you want, etc.

 

If you think I should add delays to any of these situations, which would be the ones to do it to?

 

Thanks,

 

-Taner

[TanerH]

Actually, after thinking about this some more, I recall that the "all off" happened just as I hit the "back of the house off" button on the KPL...

 

The "all on" happened when (I think) a program ran that turned on a light that would have triggered an "on" for one other KPL buttons (scene)

[gviliunas]

I have experienced the random all-ON problem for several years. Given the rare occurrence (about ~4 months) I haven't been able to exactly pinpoint the cause. I have learned a few things though:

 

1. The problem occurs with either ISY99 or ISY994 as controller in my system.

2. The problem has occurred with many different revisions of ISY firmware installed

3. The problem has only been observed when people are home. i.e. I have never arrived home and found all lights on.

4. The problem "seems" related to KPLs and wireless motion detectors. At one point, I had a KPL secondary button that triggered an ISY All-Off program. Sometimes, when I clicked that button, all lights would come on. I eventually replaced that KPL, the replacement KPL triggering the same program does not cause lights-on.

5. When this problem occurs, ISY seems overwhelmed with the massive status changed going on in the house. After the event occurs, most light status have not been updated by ISY until I manually run the global query program. Also, there is no meaningful log information captured by ISY.

 

4a. For me, the problem now seems most related to motion detectors with older batteries. Two nights ago, the problem occurred just as a motion detector was triggered. There is an older battery installed. I haven't replaced the battery but the problem has not re-occurred over the past 2 days so the low battery idea is just a guess at this point.

 

I don't query my motion detectors but this one is directly linked to a Switchlinc relay and set to send only On commands. ISY listens for this and triggers a countdown timer to turn off the light if no further motion is detected.

The status of the Switchlinc being ON triggers an ISY 1st floor light status program to light a KPL button. Lighting this KPL button through a scene is not controlled directly by the motion detector status though.

 

Troubleshooting tool / method needed:

Is there a way to program ISY to trap occurrences of group 255 (All ON) being sent by Insteon devices within the network? If ISY could do this, the ALL ON problem could be easy to troubleshoot.

 

Greg

[Michel Kohanim]

Hi Taner and Greg,

 

I can definitively tell you that the problem is that INSTEON messages are corrupted in transit and then translated as On by any devices that's on the path (from PLM). What you need to do is:

If you have any programs that change the status of a backlight button on KPL, make sure they are no loops or make sure there are delays between each activation.

If those buttons are activated because of a previous event form another device, such as a motion sensor, make sure there's a delay between the program becoming true and the KPL backlight scene being activated

 

The above two will dramatically reduce the number of occurrences (reducing collision between traffic going on and traffic coming into the PLM).

 

With kind regards,

Michel

[gviliunas]

Thanks for your suggestions Michel!

 

I don't think I have anything causing loops but I added a 3-second delay in each program that causes a KPL secondary button to change state. I use these programs to monitor all devices in different areas of the house and light KPL secondary buttons when any device in an area is on.

 

I had been suspecting that motion detectors were sending group 255 commands (without a shred of proof) but yes, my KPL status programs also could be the cause. Thanks for your help. Given the very low frequency of occurrence (~4mo), I don't expect to know for several months if this solved my problem.

 

Greg

[Michel Kohanim]

Hi Greg, thanks. Please do keep us posted.

 

With kind regards,

Michel

[TanerH]

Oops, I thought I had replied to this thread, but perhaps I never actually submitted it!

 

Anyway, I've gone ahead and added a 1sec delay to the programs that adjust any of the button scenes on my KPLs... we'll see if that helps prevent this from happening again.

 

It had only happened once each way (ON and OFF) for me in the last several months, so it might have been some other type of gremlins (since I have no motion sensors)

 

So... one strange thing.. this house still has an old Lutron RadioRA system in it, and when the "all off" event happened across all of my Insteon devices, it seemed to also turn off a couple of the Lutron-controlled lights. Now, I'm not convinced that happened at the same time (there are some light switches that remain on at all times, so I went to check those when the all-off happened, and one of them was off), and I don't even think Lutron and Insteon even operate in the same frequency spectrum, but I figured I would at least mention this as well "just in case". This might just be a coincidence (or otherwise a red herring).

 

Oh, one related question I have: Is there a way to completely disable/prevent an "ALL ON" event from being broadcast by the PLM? I have a few outlets/switches that really should not be turned on "accidentally" if possible... or at least, shouldn't possibly be "left on" for a long time.

 

If not, I guess I can write some "watchdog" programs to ensure these don't stay on for super-long.

 

Thoughts?

 

-Taner

[LeeG]

"Is there a way to completely disable/prevent an "ALL ON" event from being broadcast by the PLM?"

 

No, even if it was specifically sent to the PLM as a serial command to send. This sounds like something different from an explicit send of an ALL ON through the PLM.

[TanerH]

"Is there a way to completely disable/prevent an "ALL ON" event from being broadcast by the PLM?"

 

No, even if it was specifically sent to the PLM as a serial command to send.

Ahh, ok. I'll add a couple watchdog programs, then

 

This sounds like something different from an explicit send of an ALL ON through the PLM.

Yeah, I agree it's most likely that packets are getting corrupted on the wire and misinterpreted by the devices.

 

I have a feeling this might have to do with line noise generated by my AC units... but I have no idea how to filter/isolate those. Are there any line filters that work at high-current 220? (~60A)

 

-Taner

[arw01]

Nothing by brand comes to mind, but a call the a commercial electrical distribution company, like Consolidated Electrical Distributors, might ferret up something. I suspect some manufacturing companies would need something like this for their robotics or big welders. I have worked at several places that the company would brown out when a welder was fired up!

[Smile4yourself]

If this happened at my home, i'd be looking at my access logs to see if someone gained remote access to the controller.

 

There was an article i saw posted by google alerts under Insteon Security where the reporter had searched on google, found a list of smart homes in australia and broke in from his livingroom, turning lights on and off etc...

 

http://www.forbes.com/sites/kashmirhill ... omes-hack/

 

Best wishes

[Michel Kohanim]

Hello Smile4yourself,

 

We have already gone through the Access Logs and this issue is not at all related to a remote hacker. This is very much related to ISNTEON signals being "misinterpreted" as all on/all off in very special cases that include KPLs and Motion Sensors in the mix.

 

With kind regards,

Michel

[andrewm]

I just had this happen to me as well. In my case all my lights went on, my garage door opened, my ceiling fans went on full blast, and my sprinklers turned on. And being that this was 1am, my wife got pretty pissed! Luckily I was here and still awake when it happened.

 

Reviewing the logs does show a motion event outside the house at the time this happened - this turned on the scene for the garage lights, which includes in the scene a KPL button indicator.

Garage / Garage Motion Sensor                 Status            100%   Wed 2014/09/24 12:47:33 AM    System
Scene:Outside / Sc: Outside Garage Lights     On                 255   Wed 2014/09/24 12:47:33 AM    Program
Living Room / LR - Thermostat                 Status              76   Wed 2014/09/24 12:47:34 AM    System
Scene:Outside / Sc: Outside Garage Lights     On                 255   Wed 2014/09/24 12:47:34 AM    Program
Family Room / FR - Thermostat                 Thermostat  Mode   Off   Wed 2014/09/24 12:47:35 AM    System
Family Room / FR - Thermostat                 Fan Setting       Auto   Wed 2014/09/24 12:47:35 AM    System
Garage / Garage Door-Sensor                   Status            100%   Wed 2014/09/24 12:47:35 AM    System
Scene:Garage / Sc: Garage Door Status         On                 255   Wed 2014/09/24 12:47:35 AM    Program
Living Room / LR - Thermostat                 Thermostat Mode    Off   Wed 2014/09/24 12:47:38 AM    System
Living Room / LR - Thermostat                 Fan Setting       Auto   Wed 2014/09/24 12:47:38 AM    System
Scene:Outside / Sc: Outside Front Porch       On                 255   Wed 2014/09/24 12:47:39 AM    Program

There are no loops, but it looks like the scene did get initiated twice within a second for some unknown reason. This would appear to be consistent with other reports of this happening. In my case this is the first time it's happened in 7 years of having an ISY, so hopefully it will remain a rare occurence. However I did replace a failed PLM a week ago, and just this last weekend I also replaced the KPL in question, because of difficulty syncing the link database with the ISY; so maybe there is a connection between these changes and the AllOn event?

 

The fact the AllOn event is possible and can't be disabled on a per-device basis, is bad design by Insteon, given the network is not 100% reliable, and potentially very dangerous. The fact the event can occur accidentally casts a pall on Insteon. The really bad thing is I don't see a good way to counteract it - which is not good news when it's performing control operations such as opening doors and turning on sprinklers. The reason being that ISY had no indication that (eg) the sprinklers were turned on, so couldn't trigger a counteracting response - even my watchdogs / guards wouldn't work since all programs are bypassed. I presume the reason ISY had no indication was the network must have been flooded with status messages, though I don't see evidence of that in the log.

 

So my actions will be:

• Re-program the motion detector processing to guard against a new response when an existing response is in progress.
• Replace the sprinker control device with one that has momentary-on capability (where momentary = "60 seconds") rather than the on/off state being under control of ISY. Longer ON periods can be handled by an ISY program, but accidental ON's will only result in 60 seconds run-time. Tbh I'd actually been meaning to do this for a while

My questions / suggestions are:

• Can ISY not see these AllOn messages on the network, and create appropriate triggers for them? Eg if I could trigger a program based on an AllOn event, then at least I could initiate an automated recovery procedure.
• Are any devices more susceptible than others to creating these bad events? Eg newer / older devices?

- Andrew

[Teken]

 

I just had this happen to me as well. In my case all my lights went on, my garage door opened, my ceiling fans went on full blast, and my sprinklers turned on. And being that this was 1am, my wife got pretty pissed! Luckily I was here and still awake when it happened.

 

Reviewing the logs does show a motion event outside the house at the time this happened - this turned on the scene for the garage lights, which includes in the scene a KPL button indicator.

Garage / Garage Motion Sensor                 Status            100%   Wed 2014/09/24 12:47:33 AM    System
Scene:Outside / Sc: Outside Garage Lights     On                 255   Wed 2014/09/24 12:47:33 AM    Program
Living Room / LR - Thermostat                 Status              76   Wed 2014/09/24 12:47:34 AM    System
Scene:Outside / Sc: Outside Garage Lights     On                 255   Wed 2014/09/24 12:47:34 AM    Program
Family Room / FR - Thermostat                 Thermostat  Mode   Off   Wed 2014/09/24 12:47:35 AM    System
Family Room / FR - Thermostat                 Fan Setting       Auto   Wed 2014/09/24 12:47:35 AM    System
Garage / Garage Door-Sensor                   Status            100%   Wed 2014/09/24 12:47:35 AM    System
Scene:Garage / Sc: Garage Door Status         On                 255   Wed 2014/09/24 12:47:35 AM    Program
Living Room / LR - Thermostat                 Thermostat Mode    Off   Wed 2014/09/24 12:47:38 AM    System
Living Room / LR - Thermostat                 Fan Setting       Auto   Wed 2014/09/24 12:47:38 AM    System
Scene:Outside / Sc: Outside Front Porch       On                 255   Wed 2014/09/24 12:47:39 AM    Program

There are no loops, but it looks like the scene did get initiated twice within a second for some unknown reason. This would appear to be consistent with other reports of this happening. In my case this is the first time it's happened in 7 years of having an ISY, so hopefully it will remain a rare occurence. However I did replace a failed PLM a week ago, and just this last weekend I also replaced the KPL in question, because of difficulty syncing the link database with the ISY; so maybe there is a connection between these changes and the AllOn event?

 

The fact the AllOn event is possible and can't be disabled on a per-device basis, is bad design by Insteon, given the network is not 100% reliable, and potentially very dangerous. The fact the event can occur accidentally casts a pall on Insteon. The really bad thing is I don't see a good way to counteract it - which is not good news when it's performing control operations such as opening doors and turning on sprinklers. The reason being that ISY had no indication that (eg) the sprinklers were turned on, so couldn't trigger a counteracting response - even my watchdogs / guards wouldn't work since all programs are bypassed. I presume the reason ISY had no indication was the network must have been flooded with status messages, though I don't see evidence of that in the log.

 

So my actions will be:

• Re-program the motion detector processing to guard against a new response when an existing response is in progress.
• Replace the sprinker control device with one that has momentary-on capability (where momentary = "60 seconds") rather than the on/off state being under control of ISY. Longer ON periods can be handled by an ISY program, but accidental ON's will only result in 60 seconds run-time. Tbh I'd actually been meaning to do this for a while

My questions / suggestions are:

• Can ISY not see these AllOn messages on the network, and create appropriate triggers for them? Eg if I could trigger a program based on an AllOn event, then at least I could initiate an automated recovery procedure.
• Are any devices more susceptible than others to creating these bad events? Eg newer / older devices?

- Andrew

 

 

What does the physical sticker indicate this new PLM as? Also, what firmware does the ISY say it has in it?

[andrewm]

What does the physical sticker indicate this new PLM as? Also, what firmware does the ISY say it has in it?

ISY is 4.2.10

PLM Sticker says: 2413S, v1.C, 1425

[Brian H]

v1.C is the hardware version and 1425 is a production code.

 

What firmware is reported for the PLM?

 

Tools.

Diagnostics

PLM Info/Status.

Should show the six digit Insteon address and firmware version. Along with a connected indication.

[andrewm]

What firmware is reported for the PLM?

It says "v9B"

 

- Andrew

[Cartwrig]

Nothing by brand comes to mind, but a call the a commercial electrical distribution company, like Consolidated Electrical Distributors, might ferret up something. I suspect some manufacturing companies would need something like this for their robotics or big welders. I have worked at several places that the company would brown out when a welder was fired up!

 

I know your post is 1.5 years old; but, maybe this will still help you, or someone else reading here.

 

I had a client's home with a 120vac (or was it 220vac) variable speed well-pump controller.  The idea being that if the house needed more water the pump would speed up and slow down.  The input was standard electrical, the output was 3-phase, and operated like a 3 winding servo controller.  It created ENORMOUS noise on the powerline in the house, every time it kicked on. 

 

Took me a while to figure out that was what was creating the noise in the house (well, also because lots of other stuff was too:  Samsung TV's, PC switching power supplies, some appliances in the kitchen, etc.).  So, after putting the plug in filters on all tv's and PC's, and then the large 2-3 gang box brown filters from Smarthome.com on power wires coming out of the circuitbreaker panel and going to some of the appliances in the kitchen, then I turned to solving the variable speed well-pump controller.

 

Someone, and I don't now remember who, pointed me in 2008 to Filter Concepts, Inc.  Where I bought an XF50 Filter.  It was tuned to block either 120KHz (X10 frequencies) and/or also to 131KHz (Insteon frequencies).  It was expensive.   You can get the pdf here:  http://www.intesh.com/insteon/XF50_Application_7020-11.pdf .

 

It's rated at:  

   50Amps

   125/250VAC.

 

Enjoy,

- Rob

[Cartwrig]

I know your post is 1.5 years old; but, maybe this will still help you, or someone else reading here.

 

I had a client's home with a 120vac (or was it 220vac) variable speed well-pump controller.  The idea being that if the house needed more water the pump would speed up and slow down.  The input was standard electrical, the output was 3-phase, and operated like a 3 winding servo controller.  It created ENORMOUS noise on the powerline in the house, every time it kicked on. 

 

Took me a while to figure out that was what was creating the noise in the house (well, also because lots of other stuff was too:  Samsung TV's, PC switching power supplies, some appliances in the kitchen, etc.).  So, after putting the plug in filters on all tv's and PC's, and then the large 2-3 gang box brown filters from Smarthome.com on power wires coming out of the circuitbreaker panel and going to some of the appliances in the kitchen, then I turned to solving the variable speed well-pump controller.

 

Someone, and I don't now remember who, pointed me in 2008 to Filter Concepts, Inc.  Where I bought an XF50 Filter.  It was tuned to block either 120KHz (X10 frequencies) and/or also to 131KHz (Insteon frequencies).  It was expensive.   You can get the pdf here:  http://www.intesh.com/insteon/XF50_Application_7020-11.pdf .

 

It's rated at:  

   50Amps

   125/250VAC.

 

Enjoy,

- Rob

 

As I recall, it was a completely custom built device.  I think they already had one built for another client that never had it shipped.  So, I wound up being able to buy it.  If any of you wind up needing such a thing, you'll probably have to have one built for you.   I thought I was going to need another on the homes elevator motor; but, it ran so infrequently that I decided not to worry about it.

[Cartwrig]

So, I have a bunch of Program that set 2, 3, or 4 different "Scenes" using a Fast On (because they have CFL's in those ceiling can zones).  

After reading a bunch of the notes above, I've gone in and put a 3 second WAIT before a Scene is "set", just after a keypad or other device just transmitted.

 

My big question:  Should I be putting 3 second WAITs before each consecutive "Set Scene"?  That way each Insteon Transmission has time to do "clean-up", then the house wide system can move on to the next Set Scene?  

2nd big question:  I had always thought the Insteon had the proper CSMA (carrier sense multiple access) "attempt and back-off" functionality built into each device, such that devices would not step on top of each other.  Or, if they did, and had a collision, they would retry using a proper randomized back-off timer.....just like a zillion other technologies.   Am I wrong on this?

[LeeG]

It should not be necessary to Wait 3 seconds between Set Scene statements.

 

It is possible for conflicts to happen between devices. Devices such as a Motion Sensor, being battery powered and RF only, can have issues with other devices. The PLM can start a command sequence at the same time the Motion Sensor is progressing with its Scene On/Off. That is why it may be necessary to Wait 1 or 2 seconds before a motion sensor triggered Program issues its first Set Scene.

 

Device A will not physically step on device B message if aware of the message. Insteon will terminate a Scene if two Scenes are running at once. This aspect of Insteon has always been true and is covered in the latest insteondetails.pdf document on insteon.com