peekay Posted January 19, 2023 Share Posted January 19, 2023 I'm having issues with my ISY994 not being able to consistently communicate with my Insteon devices. It started when I put up Christmas decorations, which included pulling an old insteon outdoor plug from storage, and adding 7-10 new SonoffS31 wifi switches (running Tasmota and not on the ISY994). Initially it was just a couple devices and only occassionally... including the outdoor plug, but over the course of 2 months it has gotten to the point where the system is unusable. Some things go on but not off, some occassionally go on or off, but not consistently, and every night all insteon devices shut off at once. The serial powerline modem is about 4 years old, and the devices range from maybe 8-10 years old to a few months. Troubleshooting steps: Unplugged all Christmas decorations and controls. No impact on operations. Re plugged in all the SonoffS31 controls. Moved ISY and powerline modem to different outlets and circuits in the house. No impact on operations. Deleted and re-added a device via the admin console. Potentially improved operation, but still not reliable. Unplugged the second device that showed errors in error log (see 1 below). No impact on operations. Additional debugging: Reviewed log and identified patterns; log shows initial error with outdoor plug, and consistent pattern to what I was experiencing with additional error rate over time. Checked ISY insteon device communications count. Count was consistent (91 devices) over 10 tries. Random thoughts: Time of day impact seems to be significant; it is better during the day and worse at night. House is pretty big, but no two insteon devices are further than 20' away from another one. A number of the devices listed are motion sensors that aren't really used (not reliable enough), but are still active. House has a main panel and two sub panels, along with Enphase solar microinverters connected at main panel. I do not believe any neighbors use Insteon. Electrically, closest neighbor is about 120 line feet away, next-closest is about 320 line-feet away. Transformer is shared across ~25 homes. Air conditioning doesn't run during any of these issues and there are only two small motors that run (water feature pump and hot water circulator) during the day. PLM is not plugged into a surge supressor, and there are no whole-house surge suppressors. I am now at a complete loss on what to try. The PLMs are out of stock which would be my next debugging strategy. Thanks for the help!! Link to comment
peekay Posted January 19, 2023 Author Share Posted January 19, 2023 Just a clarification-- I call it "unreliable" in that I need to initiate the control 3-5 times for it to have an 80% chance of success. Link to comment
kclenden Posted January 19, 2023 Share Posted January 19, 2023 (edited) 1 hour ago, peekay said: The PLMs are out of stock which would be my next debugging strategy. Here are some things you can try: In the Admin Console start the Event Viewer (Tools->Diagnostics->Event Viewer). Set the Level to "3. Device communication events". Now try commanding devices from the Admin Console. Look in the Event Viewer log for "Std-Direct-Ack" lines. This is an acknowledgement from the device to the ISY (via the PLM). Note the "Max Hops" and "Hops Left" values. The higher the "Hops Left" value the better. It's a measure of how many times a message was repeated by another device as it traversed from the sending device to the receiving device (each repeat causing hops left to go down one). The more times a message has to be repeated, the harder the network it working to get the message to its intended destination which generally means there is some interference on the line. Try connecting a device to the same circuit as the PLM and repeat #1. You should see "Max Hops" and "Hops Left" always being equal. Some people wire switches to a three-prong plug and put the PLM and switch on the same power strip. Shutoff all circuit breakers but the one for the PLM. Turn on a single circuit breaker for another device. Try commanding that device as in #1 above. If communication is good, then turn on another circuit breaker. Test and repeat until turning on a circuit breaker significantly degrades communication Edited January 19, 2023 by kclenden 2 Link to comment
peekay Posted January 20, 2023 Author Share Posted January 20, 2023 45 minutes ago, kclenden said: In the Admin Console start the Event Viewer... Tried that much and not much remarkable during the day. I think the most I had to repeat a command was 4-5x for the farthest devices. Not quite sure how to read the balance of the events and errors Closest device Thu 01/19/2023 14:18:08 : [INST-TX-I1 ] 02 62 57 65 64 0F 11 4D Thu 01/19/2023 14:18:08 : [INST-ACK ] 02 62 57.65.64 0F 11 4D 06 LTONRR (4D) Thu 01/19/2023 14:18:08 : [INST-SRX ] 02 50 57.65.64 40.53.13 2F 11 4D LTONRR (4D) Thu 01/19/2023 14:18:08 : [Std-Direct Ack] 57.65.64-->ISY/PLM Group=0, Max Hops=3, Hops Left=3 Thu 01/19/2023 14:18:08 : [D2D EVENT ] Event [57 65 64 1] [ST] [77] uom=100 prec=0 Thu 01/19/2023 14:18:08 : [ 57 65 64 1] ST 77 (uom=100 prec=0) Thu 01/19/2023 14:18:15 : [INST-TX-I1 ] 02 62 57 65 64 0F 13 00 Thu 01/19/2023 14:18:15 : [INST-ACK ] 02 62 57.65.64 0F 13 00 06 LTOFFRR(00) Thu 01/19/2023 14:18:16 : [INST-SRX ] 02 50 57.65.64 40.53.13 2B 13 00 LTOFFRR(00) Thu 01/19/2023 14:18:16 : [Std-Direct Ack] 57.65.64-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:18:16 : [D2D EVENT ] Event [57 65 64 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:18:16 : [ 57 65 64 1] ST 0 (uom=100 prec=0) Least reliable Far Device Thu 01/19/2023 14:28:09 : [INST-TX-I1 ] 02 62 54 A0 07 0F 13 00 Thu 01/19/2023 14:28:09 : [INST-ACK ] 02 62 54.A0.07 0F 00 FF 06 (FF) Thu 01/19/2023 14:28:10 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 00 FF (FF) Thu 01/19/2023 14:28:10 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:28:22 : [INST-TX-I1 ] 02 62 54 A0 07 0F 13 00 Thu 01/19/2023 14:28:35 : [INST-TX-I1 ] 02 62 54 A0 07 0F 13 00 Thu 01/19/2023 14:28:35 : [INST-ACK ] 02 62 54.A0.07 0F 13 00 06 LTOFFRR(00) Thu 01/19/2023 14:28:37 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 13 00 LTOFFRR(00) Thu 01/19/2023 14:28:37 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:28:37 : [D2D EVENT ] Event [54 A0 7 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:28:37 : [ 54 A0 7 1] ST 0 (uom=100 prec=0) Thu 01/19/2023 14:29:09 : [INST-TX-I1 ] 02 62 54 A0 07 0F 11 FF Thu 01/19/2023 14:29:24 : [INST-TX-I1 ] 02 62 54 A0 07 0F 11 FF Thu 01/19/2023 14:29:25 : [INST-ACK ] 02 62 54.A0.07 11 FF 00 02 62 54 A0 07 0F 11 FF 06 02 50 54 A0 07 40 (00) Thu 01/19/2023 14:29:29 : [D2D EVENT ] Event [54 A0 7 1] [ERR] [1] uom=0 prec=-1 Thu 01/19/2023 14:29:29 : [ 54 A0 7 1] ERR 1 Thu 01/19/2023 14:29:37 : [INST-TX-I1 ] 02 62 54 A0 07 0F 12 00 Thu 01/19/2023 14:29:50 : [INST-TX-I1 ] 02 62 54 A0 07 0F 12 00 Thu 01/19/2023 14:29:50 : [INST-ACK ] 02 62 54.A0.07 0F 12 00 06 LTON-F (00) Thu 01/19/2023 14:29:50 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 12 00 LTON-F (00) Thu 01/19/2023 14:29:50 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:29:50 : [D2D EVENT ] Event [54 A0 7 1] [ERR] [0] uom=0 prec=-1 Thu 01/19/2023 14:29:50 : [ 54 A0 7 1] ERR 0 Thu 01/19/2023 14:29:50 : [D2D EVENT ] Event [54 A0 7 1] [ST] [255] uom=100 prec=0 Thu 01/19/2023 14:29:50 : [ 54 A0 7 1] ST 255 (uom=100 prec=0) Thu 01/19/2023 14:29:57 : [INST-TX-I1 ] 02 62 54 A0 07 0F 14 00 Thu 01/19/2023 14:29:57 : [INST-ACK ] 02 62 54.A0.07 0F 14 00 06 LTOFF-F(00) Thu 01/19/2023 14:29:58 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 14 00 LTOFF-F(00) Thu 01/19/2023 14:29:58 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:29:58 : [D2D EVENT ] Event [54 A0 7 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:29:58 : [ 54 A0 7 1] ST 0 (uom=100 prec=0) I'll have to wait until tonight to try to get better data and likely tomorrow before I can power cycle everything. Link to comment
kclenden Posted January 20, 2023 Share Posted January 20, 2023 (edited) 1 hour ago, peekay said: I'll have to wait until tonight to try to get better data and likely tomorrow before I can power cycle everything. Everything in the first log segment looks normal. "Hops Left" is either 3 or 2 which seems to indicate good communication between PLM and Device. Here's an interpretation of the log segment so you have an idea of what things mean: Thu 01/19/2023 14:18:08 : [INST-TX-I1 ] 02 62 57 65 64 0F 11 4D 02 = Start of Command 62 = Send Message 57 65 64 = Address of Device the Message is being sent to OF = Message Flags (OF means Direct Standard Message, Hops Left=3, Max Hops=3) 11 = Command for Device to Execute (11 means GO ON) 4D = LEVEL (Level is in HEX 00-FF so 4D/FF which is 77/256 is 30%) Thu 01/19/2023 14:18:08 : [INST-ACK ] 02 62 57.65.64 0F 11 4D 06 LTONRR (4D) PLM echos back to ISY the command and appends 06 to indicate that the message was successfully sent Thu 01/19/2023 14:18:08 : [INST-SRX ] 02 50 57.65.64 40.53.13 2F 11 4D LTONRR (4D) 02 = Start of Command 50 = Message Received 57 65 64 = Address of Device the Message is from 40 53 13 = Address of Device the Message is being sent 2F = Message Flags (2F means ACK Direct Message, Hops Left=3, Max Hops=3) 11 = Command for Device to Execute (11 means GO ON) 4D = LEVEL (Level is in HEX 00-FF so 4D/FF which is 77/256 is 30%) Thu 01/19/2023 14:18:08 : [Std-Direct Ack] 57.65.64-->ISY/PLM Group=0, Max Hops=3, Hops Left=3 Thu 01/19/2023 14:18:08 : [D2D EVENT ] Event [57 65 64 1] [ST] [77] uom=100 prec=0 Thu 01/19/2023 14:18:08 : [ 57 65 64 1] ST 77 (uom=100 prec=0) Shows ISY interpretaton of received message ---------- Thu 01/19/2023 14:18:15 : [INST-TX-I1 ] 02 62 57 65 64 0F 13 00 02 = Start of Command 62 = Send Message 57.65.64 = Address of Device the Message is being sent to OF = Message Flags (OF means Direct Standard Message, Hops Left=3, Max Hops=3) 13 = Command for Device to Execute (13 means GO OFF) 00 = Means nothing just default for command 13 Thu 01/19/2023 14:18:15 : [INST-ACK ] 02 62 57.65.64 0F 13 00 06 LTOFFRR(00) PLM echos back to ISY the command and appends 06 to indicate that the message was successfully sent Thu 01/19/2023 14:18:16 : [INST-SRX ] 02 50 57.65.64 40.53.13 2B 13 00 LTOFFRR(00) 02 = Start of Command 50 = Message Received 57 65 64 = Address of Device the Message is from 40 53 13 = Address of Device the Message is being sent 2B = Message Flags (2B means ACK Direct Message, Hops Left=2, Max Hops=3) 13 = Command for Device to Execute (13 means GO OFF) 00 = Means nothing just default for command 13 Thu 01/19/2023 14:18:16 : [Std-Direct Ack] 57.65.64-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:18:16 : [D2D EVENT ] Event [57 65 64 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:18:16 : [ 57 65 64 1] ST 0 (uom=100 prec=0) Shows ISY interpretaton of received message For the second log segment. I didn't break out the meaning of each log line since they could be interpreted using the same method as above. "Hops Left" is always 2. This indicates good communication between PLM and Device, but the fact that it's always 2 could mean that the device is always depending on another device to relay its messages, or could just be a function of limited data. If it's the former, it could be a Device that is on a different power leg than the PLM, or an RF only device that is out of RF range of the PLM and needs a power line device to receive its message and forward it. Here are some comments on things that look strange. I also ignored the first four lines since they seemed to be two different commands colliding in time: Thu 01/19/2023 14:28:22 : [INST-TX-I1 ] 02 62 54 A0 07 0F 13 00 It doesn't appear that the ISY received an acknowledgement from the PLM ---------- Thu 01/19/2023 14:28:35 : [INST-TX-I1 ] 02 62 54 A0 07 0F 13 00 Thu 01/19/2023 14:28:35 : [INST-ACK ] 02 62 54.A0.07 0F 13 00 06 LTOFFRR(00) Thu 01/19/2023 14:28:37 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 13 00 LTOFFRR(00) Thu 01/19/2023 14:28:37 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:28:37 : [D2D EVENT ] Event [54 A0 7 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:28:37 : [ 54 A0 7 1] ST 0 (uom=100 prec=0) Log indicates OFF was successful ---------- Thu 01/19/2023 14:29:09 : [INST-TX-I1 ] 02 62 54 A0 07 0F 11 FF It doesn't appear that the ISY received an acknowledgement from the PLM ---------- Thu 01/19/2023 14:29:24 : [INST-TX-I1 ] 02 62 54 A0 07 0F 11 FF Thu 01/19/2023 14:29:25 : [INST-ACK ] 02 62 54.A0.07 11 FF 00 02 62 54 A0 07 0F 11 FF 06 02 50 54 A0 07 40 (00) The PLM's acknowledgement back to the ISY seems to be corrupted. It seems to be a partial acknowledgement, a full acknowledgement, and a partial received message all crammed together. Thu 01/19/2023 14:29:29 : [D2D EVENT ] Event [54 A0 7 1] [ERR] [1] uom=0 prec=-1 Thu 01/19/2023 14:29:29 : [ 54 A0 7 1] ERR 1 The ISY recognizes the acknowledgement as an error ---------- Thu 01/19/2023 14:29:37 : [INST-TX-I1 ] 02 62 54 A0 07 0F 12 00 Like above, it doesn't appear that the ISY received an acknowledgement from the PLM ---------- Thu 01/19/2023 14:29:50 : [INST-TX-I1 ] 02 62 54 A0 07 0F 12 00 Command 12 is a Fast ON Thu 01/19/2023 14:29:50 : [INST-ACK ] 02 62 54.A0.07 0F 12 00 06 LTON-F (00) Correct acknowledgement from the PLM Thu 01/19/2023 14:29:50 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 12 00 LTON-F (00) Thu 01/19/2023 14:29:50 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:29:50 : [D2D EVENT ] Event [54 A0 7 1] [ERR] [0] uom=0 prec=-1 Thu 01/19/2023 14:29:50 : [ 54 A0 7 1] ERR 0 Thu 01/19/2023 14:29:50 : [D2D EVENT ] Event [54 A0 7 1] [ST] [255] uom=100 prec=0 Thu 01/19/2023 14:29:50 : [ 54 A0 7 1] ST 255 (uom=100 prec=0) Log seems to indicate Fast ON was successful. Not sure what ERR 0 means. ---------- Thu 01/19/2023 14:29:57 : [INST-TX-I1 ] 02 62 54 A0 07 0F 14 00 Command 14 is a Fast OFF Thu 01/19/2023 14:29:57 : [INST-ACK ] 02 62 54.A0.07 0F 14 00 06 LTOFF-F(00) Thu 01/19/2023 14:29:58 : [INST-SRX ] 02 50 54.A0.07 40.53.13 2B 14 00 LTOFF-F(00) Thu 01/19/2023 14:29:58 : [Std-Direct Ack] 54.A0.07-->ISY/PLM Group=0, Max Hops=3, Hops Left=2 Thu 01/19/2023 14:29:58 : [D2D EVENT ] Event [54 A0 7 1] [ST] [0] uom=100 prec=0 Thu 01/19/2023 14:29:58 : [ 54 A0 7 1] ST 0 (uom=100 prec=0) Log indicates Fast OFF was successful Given that there are log entries that seem to indicate the PLM didn't acknowledge messages from the ISY, and log entries that seem to indicate the PLM sent corrupted data to the ISY, my guess is that your PLM is failing. I suppose it could also be that the RS232 port on the ISY is failing, but I've never read of that happening. Edited January 20, 2023 by kclenden 1 1 Link to comment
peekay Posted January 20, 2023 Author Share Posted January 20, 2023 @kclendenThank you so much for the detailed breakdown! I'll try to watch for the corrupted messages tonight and see what I can glean from the data. Unfortunately though it looks like this will end up being the nail in the coffin for my Insteon system unless the PLMs come in stock soon. Link to comment
kclenden Posted January 20, 2023 Share Posted January 20, 2023 6 hours ago, peekay said: Unfortunately though it looks like this will end up being the nail in the coffin for my Insteon system unless the PLMs come in stock soon. There are PLM repair services available online. I think Insteon itself even offers this service. That would be much cheaper than replacing your entire setup. 🙂 Additionally, the last I heard new PLMs were slated to be available by March, I think. Though there will probably be a run on them right off the bat since they've been unavailable for so long. 1 Link to comment
kclenden Posted January 20, 2023 Share Posted January 20, 2023 6 hours ago, peekay said: Thank you so much for the detailed breakdown! I'll try to watch for the corrupted messages tonight and see what I can glean from the data. You'll also want to look for instances where you see an [INST-TX-I1] in the Event Viewer, but don't immediately see a following [INST-ACK] with the command echoed back with an "06" appended. In these instances, the device shouldn't have reacted to your command from the Admin Console because the PLM never received the command from the ISY to send onto the device. While I really suspect your PLM, I suppose you might try disconnecting the cable between the PLM and the ISY (looks like a network cable but isn't) and reconnecting it just in case it is somehow loose. Link to comment
MrBill Posted January 20, 2023 Share Posted January 20, 2023 12 hours ago, peekay said: Unfortunately though it looks like this will end up being the nail in the coffin for my Insteon system unless the PLMs come in stock soon. The new Insteon says March for new PLMs. The new Insteon company also offers PLM repair services. There's also this repair service operate through ebay. the ebay guy also posts in this forum. 1 Link to comment
peekay Posted January 21, 2023 Author Share Posted January 21, 2023 (edited) 10 hours ago, MrBill said: The new Insteon says March for new PLMs. The new Insteon company also offers PLM repair services. There's also this repair service operate through ebay. the ebay guy also posts in this forum. If it lasts me until March then I can get a new one... the problem is Insteon won't front you a refurb before you send in your unit so you are down until it is repaird. So, highest Wife Approval Factor wins... Edited January 21, 2023 by peekay Link to comment
MrBill Posted January 21, 2023 Share Posted January 21, 2023 13 hours ago, peekay said: If it lasts me until March then I can get a new one... the problem is Insteon won't front you a refurb before you send in your unit so you are down until it is repaird. So, highest Wife Approval Factor wins... While more expensive I'm sure if you pay the return charges as well you can make arrangements with Nilachi Data Systems (the ebay guy, its a one man operation) to overnight the package. Still a couple days of down time, but its better than no home automation. Can't say the same for insteon proper they're a little bigger but you could certainly ask, their email queue no longer has a long delay Link to comment
xgfreon Posted January 23, 2023 Share Posted January 23, 2023 @kclenden Thanks for the log breakdown. Often wondered this myself when i had previous comm failures. Link to comment
Brian H Posted January 23, 2023 Share Posted January 23, 2023 Just got an email from UDI. There is a 20% discount on PLM repairs through the new Insteon site. With a discount code in the information page of Insteon.com 2 Link to comment
larryllix Posted January 24, 2023 Share Posted January 24, 2023 On 1/23/2023 at 8:25 AM, Brian H said: Just got an email from UDI. There is a 20% discount on PLM repairs through the new Insteon site. With a discount code in the information page of Insteon.com Leave it to UDI to get their users a discount !!!! Link to comment
peekay Posted March 2, 2023 Author Share Posted March 2, 2023 (edited) Well, back on this. I got the PLM refurbished by Insteon, and there is no change in the behavior. Not sure if they just replace the bad capacitor that used to be problematic or if they do more work. When I was updating the modem from the ISY there were no surprising errors-- just stuff that had been removed from the system, plus one exception of an active device when refreshing status. (The two adjacent Insteon devices, physically and electrically, were fine.) Then, as before the PLM stopped responding. Edited March 2, 2023 by peekay Link to comment
peekay Posted March 2, 2023 Author Share Posted March 2, 2023 When I just re-restored the PLM I got this error message: The previous time I got the "no network connection" error as well, but communications were fine over the network. Not sure if it is relevant. After the restore completed the communications is still broken. Link to comment
DennisC Posted March 2, 2023 Share Posted March 2, 2023 9 hours ago, peekay said: When I just re-restored the PLM I got this error message: The previous time I got the "no network connection" error as well, but communications were fine over the network. Not sure if it is relevant. After the restore completed the communications is still broken. Could it be your cable? Link to comment
peekay Posted March 2, 2023 Author Share Posted March 2, 2023 55 minutes ago, DennisC said: Could it be your cable? I have tried different cables, but I will open up the ISY and see if there is anything odd (corrosion) inside, and also see if I can hook it up to the Polisy instead-- forget if I still have the cable/adapter for it. Link to comment
Recommended Posts