System is completely down, and not sure where to start looking

[jim_]

I have a ISY 994/IR Pro running v.5.3.0, and today completely failed to control anything after being in service for years.  I power cycled the unit several times and the PLM.  

I noted the power LED is out, but I'm able to login to the unit via Admin console. After a power cycle, TX & RX and memory LEDs flash like normal and if you wait ... the power LED flash a few times, and then the RCV LED flashes randomly once boot is complete.   After a power cycling the ISY, the admin console show lots of errors, "Can NOT communicate with module" .... almost all modules are reporting as failed (but not all).  Those that show on line cannot be controlled via the admin console. 

I have a new ISY (purchase for my kids new house), so a swapped the power supply, no change to power LED.  Also swapped the PLM with a new one.

I've never had to diagnose a failure like this before, so any help would be appreciated 

Update 1:   received pop-up,   Error  Could not open file  /conf/UZWNDKEY.TXT  &&  Could not open config file to read /conf/UZWNDKEY.txt

Update 2:  able to control a few switches from admin console now, but most are still showing offline, seeing more Error msg pop-ups about cant write to different file names ??? 

Update 3:  no real update, did a factory reset and reload ... no change,  admin console continues to stack up failed to communicate messages upon login, I'm able to control a few switches after a reboot, but over time this degrades to only one switch on the same breaker line as the PLM.  Its like there is mass noise affecting communications, but there are no changes to hydro.  Since its Saturday morning, I reset and tested at 7am when there is no power activity from neighbours on same transformer

**** Could someone please suggest which log I should be looking or I could post to determine root cause ****

 

 

[Brian H]

The PLM has a reputation of failing. In tools, diagnostics, PLM Info/Status does it show connected and the PLM six digit ID Number?  Does it get better for a short time if you power cycle the PLM and ISY994i? Is the LED on the side of the PLM On at about the normal brightness?

When you tried the new PLM. You would have to do a restore  PLM after a power cycle of the ISY994i and PLM  or all the Insteon modules would have the old PLM 6 digit ID in them and not respond to the new PLM 6 digit ID.  The system would be busy rewriting the new PLM 6 digit ID into every module. This is for Insteon. Z-Wave doesn't use the PLM 6 digit ID.

The power supply also has been reported to fail.  5VDC to 30VDC. 1 amp probably best. The UDI store had a 12VDC 1 amp last time I looked.

Do you have Z-Wave as the files you pointed out. Seem to be Z-Wave files but probably checked even if Z-Wave is not installed.

If the power supply was flaky. It may have corrupted the SD Card. With file errors now happening.

Event Viewer in Level 3 may show some of the communications failures and successes. The Log and Error Logs may also show some information.

 

[jim_]

Thanks Brian for your reply ... here's where I'm at 

- Since the original ISY & PLM combo could not be trusted and I have a new ISY & PLM (for my kid's new house, delayed by Covid), moving to full new equipment test.  I looked through my old backups and applied a config to match the old 4.9 firmware on the new ISY, rebooted both, then run PLM update.   This old config resulted in 50% of the important modules from the DB being restored, so testing limited to important modules only.

2) Disabled all programs, and deleted modules not longer present from the config,  then started validating legacy modules with simple on / off via command menu.   50% work, 50% pop failed communications.   Some of the failed switches are recoverable by invoking factory reset, purging them from the DB and re-adding from scratch.  This appears to be working for a percentage, but not all.

3) here's an example ... simple LampLinc dimmer ... would NOT respond when connected to its original location.  Moved to Stove plugs to test both sides of supply phase.

Had to factory reset several times, finally added back as new device.   Comms fail 50% of the time, phase 2 fails more than phase 1 .... Move the module back to original outlet and ran trace ... this is where I kinda get a little lost and ask for some assistance 

You can see failures only on the OFF request from the trace ... NEW PLM ID is correct in the trace (see attached logs)

I also tried add a new out of the box 2477D ... results in failure, not sure where to translate the cause codes (see attached logs)

----------------------------------------------------------------------------------------------------------------------------------------

UPDATE:  Question,  back and forth trying different things, and noted this ... the new PLM has V2.6 software and the original has V2.0. I get more success communicating with devices on the old PLM with the new ISY ... I get major failures with the new PLM and new ISY

IS there a software version issue ? Does the ISY have to be at the latest 5.3.x to work with new PLM ?

 

 

 

 

on - off with errors - ISY-Events-Log.v4.9.0__Sat 2021.05.22 02.30.16 PM.txt adding a new device failure - ISY-Events-Log.v4.9.0__Sat 2021.05.22 02.35.36 PM.txt

[Brian H]

I have seen no posts indicating the new PLM requires 5.3.x. I have no information on what V2.6 has over the older ones. It could be possible it has revised transmitter and receiver circuits.

Are your modules Dual Band or the earlier power line only ones? 

Coupling between the incoming power lines? If you have the 2443 Access Points for coupling and they are any V2.0 or above. They are built on the same base 2413 main board and suffer the same power line issues we see in the earlier versions of the 2313S PLM.  I had a pair mess my system up until I rebuilt them. The V1.X 2443 where on a different chassis and where much better. You may want to run the built in communications tests in the Access Points, Range Extenders or Dual Band modules. To see what the tests may show.

Anything new in the home or an existing device moved to a new location? Around the time the communications started to go down hill? Some electronics generate power line noise or are signal suckers. I bought a few of those new deformable garage lights. One model has a .2uF capacitor across the power input and it severely absorbs power line signals. The other model I was trying generates so much RF garbage it can kill things like a garage door openers Rf controller key fob.

[jim_]

All my modules are dual band, no Zwave, I have a custom bridge between phases wired inside panel (which I also change on spec with no resolve) and no new AC devices connected that have not existed in the house for months. I too have those Costco deformable garage lights on a 4-way for past year with no issues. 

My experience dates back to X10 days and have had to trouble shoot many strange noise issues before, but this failure was massive. Later last night, I restored the original ISY & PLM and things are SLOWLY beginning to work again.  Many of the switch modules (not all) were failing basic function as in, you could turn lights "on" manually, but "off" either delayed / failed ... and the greater number of modules were flashing RED with errors on manual activation.   

Its like a wave of corruption travelled through all modules to point of 100% failure. 

The only notable change was temperature, we had a mini heatwave in the great white north, so everyone's AC was running on the street, but all my core equipment is kept cool and powered via a good quality UPS (which I also reset on spec as they can generate nasty harmonics when hit with power variations from the power grid)

The current resolve appears to be slowly working through deleting affected modules from ISY, factory reset them, add them back into ISY, write changes manually to all devices related to each other / scenes.   I still have no power LED on the ISY, but it seems to be working properly for now ... 

Still in restore mode for now  ... I really need to learn how to decider the logs on the ISY ... and suggested articles 

 

[Brian H]

The power LED in mine is driven directly from the ISY units power supply and a current limiting resistor.

There have been a few others posting here that their power LED was off. With no other symptoms being seen.

You may want to see if anything in the troubleshooting section of the UDI wiki. I thought I saw a section on the log entries and will see if I can find them. The UDI Cookbook also in the wiki may have something in it.

https://wiki.universal-devices.com/index.php?title=Main_Page#INSTEON_2

https://wiki.universal-devices.com/index.php?title=Main_Page

 

 

[jim_]

Well, system is 99% back with no explanation of root cause of failure.  There are still a few ghost issues where monitor indicators on KPL's are not always showing and a timed event turning off a switch not associated with a scene.  I will open the ISY and measure the volts to the power LED once everything is stable ... maybe just solder in a new LED if the original is dead.

Thanks to all for their suggestions and info 

Cheers

 

[jim_]

I spoke to soon, seems anything with timers like a motion detector are affected ... one trigger, ok .... 2 or more triggers back to back and timer (scene) is very slow or fails 

Anyone encountered timers (scenes) being delayed and root cause ?

[Brian H]

Have you tried opening a trouble ticket with UDI.

They have a good support system in place. You may want to mention this thread to them.

[bclarke01]

I'm having basically the same problem. Everything mysteriously stopped working with ISY unable to control anything and errors reported when ISY tries to query Insteon devices. Everything has worked fine for a couple of years so can't think of any recent changes that might result in this behavior. 

Power cycled both PLM and ISY and finally got things I thought working about 12:15 last night. At 2:00am, a very old program runs that shuts the house down and runs a query against the whole ISY network. All the queries returned errors and now the ISY is back to not being able to control anything. It can see changes instigated from keypads and switches, but is unable to make any changes on its own. 

It feels like there is a communication or synching problem to the PLM, but getting things working OK for awhile and then having them go awry again seems very strange. 

Yes, a ticket is opened with UDI support. I'll update this thread when I hear back. But for now, add me to the list of users experiencing this same problem.

[larryllix]

2 hours ago, bclarke01 said:

I'm having basically the same problem. Everything mysteriously stopped working with ISY unable to control anything and errors reported when ISY tries to query Insteon devices. Everything has worked fine for a couple of years so can't think of any recent changes that might result in this behavior. 

Power cycled both PLM and ISY and finally got things I thought working about 12:15 last night. At 2:00am, a very old program runs that shuts the house down and runs a query against the whole ISY network. All the queries returned errors and now the ISY is back to not being able to control anything. It can see changes instigated from keypads and switches, but is unable to make any changes on its own. 

It feels like there is a communication or synching problem to the PLM, but getting things working OK for awhile and then having them go awry again seems very strange. 

Yes, a ticket is opened with UDI support. I'll update this thread when I hear back. But for now, add me to the list of users experiencing this same problem.

If the admin console is working properly and simple programs can operate variables etc. I would try manually operating Insteon devices from the admin console. If that doesn't work I would do a link count of the PLM from admin console and if that doesn't work then buy a new PLM quickly.