ISY994i Stopped Commanding Insteon Devices

[lilyoyo1]

1 hour ago, Thomas Roach said:

Since no one else seems to have experienced this issue (or isolated it if they have), then either I abandon the system, fix it, or put on the popular band-aids every time it happens again.

Now let's talk about learning the system. Where's the source code, is it on GitHub? The schematics? What's the programming language, or the processor for that matter. Do any breakdowns of the database exist, since it looks more and more like a pointer issue? Where do I find the link structure and PLM protocols? Where's a good spot to patch in some diagnostic programs, say to get a better grasp on the actual communications instead of summaries and interpretations? How about the Portal protocols? The information I need does not appear to be readily in the wiki, nor is the wiki up to date.

And yes, that is the level of learning that I DID NOT want to get into if I could avoid it. I do not want to re-engineer the firmware. I do not want to learn about power and other control systems for components that I have a slim chance of recouping my investment costs during my remaining lifetime. I just want my lights and garage door to work, especially when I'm not home or I'm incapacitated.

 

None of what you're asking for is needed to learn or troubleshoot issues that you're having or would have. If you need that level just to use your system, you need to apply for a job with them or become a developer. Udi nor insteon is giving the keys out to anyone about their systems for it to be re-engineered.

Reading the wiki, isy cookbook and understanding the individual technology you're using is what will help you understand and troubleshoot. The insteon White pages will give you an understanding of insteon which would help in times of need. 

[Thomas Roach]

@Teken

Thanks for the list of GP tips.

It does seem SmartHome devices are considerably lacking in the quality department. That is why I have shifted more towards web-based controllers in recent years with varying degrees of success: If the Internet (or the company's servers) go down I lose monitoring and/or control, AND all these devices overload the router tables in consumer-level equipment (thus the LinkSys business-class router).

One interesting (relevant?) note: After the last occurrence, I dumped and cleared all the logs. Everything has functioned normally since that point.

 

[larryllix]

99.99% of these cases are not from ISY bugs and almost always found to be user error in understanding how event triggered programing languages operate.

Been there many time myself.

Sent using Tapatalk

[Thomas Roach]

46 minutes ago, larryllix said:

99.99% of these cases are not from ISY bugs and almost always found to be user error in understanding how event triggered programing languages operate.

Been there many time myself.

Sent using Tapatalk
 

99.99% of what cases? Cannot communicate with device cases?

And what kind of "event triggered programing languages" are you referring to? The little If-Then-Else routines of ISY programs or the Micro-C OS of the ISY itself?

I've dealt with many noise and faulty device issues over the years I've had Insteon devices, and the ISY programs are very easy to debug and test compared with other languages I've used.

What I have not seen are other instances where simply re-learning the device links resolves the problem.

 

[larryllix]

1 hour ago, Thomas Roach said:

99.99% of what cases? Cannot communicate with device cases?

And what kind of "event triggered programing languages" are you referring to? The little If-Then-Else routines of ISY programs or the Micro-C OS of the ISY itself?

I've dealt with many noise and faulty device issues over the years I've had Insteon devices, and the ISY programs are very easy to debug and test compared with other languages I've used.

What I have not seen are other instances where simply re-learning the device links resolves the problem.

 

99.99% of cases of people blaming ISY for errors.
ISY programs are event based triggers. This can be confusing for many  people and cause sneaky problems that can be hard to diagnose, especially for people with inline code language experience.

Your problem may surface again due to the above. I have had bugs in my ISY code that has taken month to resolve due to the complexity that can be created with ISY even triggering. Some take month or years to get the right combination of events to surface. ISY has no compiler or runtime errors displayed. Most other languages have progressed the development environment much more than ISY.

[Teken]

99.99% of what cases? Cannot communicate with device cases?
And what kind of "event triggered programing languages" are you referring to? The little If-Then-Else routines of ISY programs or the Micro-C OS of the ISY itself?
I've dealt with many noise and faulty device issues over the years I've had Insteon devices, and the ISY programs are very easy to debug and test compared with other languages I've used.
What I have not seen are other instances where simply re-learning the device links resolves the problem.
 

A couple things to keep in mind with respect to links. I’m going to over generalize for a moment so bare with me. In rare cases an install is too large (max hardware) and the system will not accept the inclusion of said new device.

This is limited by the 2413S PLM and what model 994 Series Controller. A pro version offers 1000 links / programs the none version quite a bit less (400?).

This can be resolved by upgrading the controller but only addresses the issue of the controller not the PLM.

So if you have say 500 devices - no dice!

Next, let’s assume the 2413S PLM is about to die. You can do a link compare (at least 3 times) to see if there is a large variance. Seeing say 30-50 links difference is something to review. Keeping in mind the link compare is not gospel so run it several times to get an idea and the house must be dead quiet when done!

In the WiKi there is a link calculator so this again will give an idea if you believe you’re nearing the PLM hardware limit.

Lastly, let’s just assume it’s the 2413S PLM. One of the indicators is missing links so when you hit restore from the controller you’re literally adding the dropped link from the PLM so it works again.

Again, this doesn’t even address the switch or any of the items I called out up above which should be the first items to review and action. Be methodical and focus on a single item and knock it off the board.

The first couple action items takes very little effort but is time intensive!

Work toward progress - Not perfection!


Sent from my iPhone using Tapatalk

[Thomas Roach]

16 minutes ago, larryllix said:

99.99% of cases of people blaming ISY for errors.
ISY programs are event based triggers. This can be confusing for many  people and cause sneaky problems that can be hard to diagnose, especially for people with inline code language experience.

Your problem may surface again due to the above. I have had bugs in my ISY code that has taken month to resolve due to the complexity that can be created with ISY even triggering. Some take month or years to get the right combination of events to surface. ISY has no compiler or runtime errors displayed. Most other languages have progressed the development environment much more than ISY.

The ISY is running If-Then-Else routines inside of its OS. There is no compiler ASAIK.

If you are talking SDK routines using ISY, then that's a different environment than I'm using.

And how does any of this affect off-on commands directly issued via the Admin Counsel?

If you have examples or links to these bugs you found difficult to resolve, perhaps that would prove instructive.

 

[Thomas Roach]

13 minutes ago, Teken said:

 

A couple things to keep in mind with respect to links. I’m going to over generalize for a moment so bare with me. In rare cases an install is too large (max hardware) and the system will not accept the inclusion of said new device.

This is limited by the 2413S PLM and what model 994 Series Controller. A pro version offers 1000 links / programs the none version quite a bit less (400?).

This can be resolved by upgrading the controller but only addresses the issue of the controller not the PLM.

So if you have say 500 devices - no dice!

Next, let’s assume the 2413S PLM is about to die. You can do a link compare (at least 3 times) to see if there is a large variance. Seeing say 30-50 links difference is something to review. Keeping in mind the link compare is not gospel so run it several times to get an idea and the house must be dead quiet when done!

In the WiKi there is a link calculator so this again will give an idea if you believe you’re nearing the PLM hardware limit.

Lastly, let’s just assume it’s the 2413S PLM. One of the indicators is missing links so when you hit restore from the controller you’re literally adding the dropped link from the PLM so it works again.

Again, this doesn’t even address the switch or any of the items I called out up above which should be the first items to review and action. Be methodical and focus on a single item and knock it off the board.

The first couple action items takes very little effort but is time intensive!

Work toward progress - Not perfection!


Sent from my iPhone using Tapatalk

 

I did not save copies of the PLM link tables.

However, the first time I ran the link table (which was before I replaced anything), it had 95 links.

The second (and last) time I ran the PLM link table (while the system seems to be fully functioning) it had 92 links.

Note that I disabled an offline device between these two counts.

This is a Pro version ISY994iz.

 

[Teken]

I did not save copies of the PLM link tables.
However, the first time I ran the link table (which was before I replaced anything), it had 95 links.
The second (and last) time I ran the PLM link table (while the system seems to be fully functioning) it had 92 links.
Note that I disabled an offline device between these two counts.
This is a Pro version ISY994iz.
 

How many Insteon devices reside in this home?


Sent from my iPhone using Tapatalk

[Thomas Roach]

13 hours ago, Teken said:

 

How many Insteon devices reside in this home?


Sent from my iPhone using Tapatalk

 

I've only got about 30 online, if you don't count details, Insteon devices converted to general web, or the drawer full of erratic/spare/defective units.

Most of my HA is done through other platforms, but they're all cloud-based.

The advantages of the ISY are the depth of programmable options, the local operation, and the time it saved me of writing my own control system on a spare Windows or Linux computer after the Insteon Hub died.

 

[larryllix]

As a cheap and easy trial you could replace your SD card with a new one. After doing this I can say it is incredibly easy and takes about 30 minutes to be fully recovered. ISY eproms contain all the necessary boot up stuff.

Sent using Tapatalk

[Thomas Roach]

19 hours ago, lilyoyo1 said:

None of what you're asking for is needed to learn or troubleshoot issues that you're having or would have. If you need that level just to use your system, you need to apply for a job with them or become a developer. Udi nor insteon is giving the keys out to anyone about their systems for it to be re-engineered.

Reading the wiki, isy cookbook and understanding the individual technology you're using is what will help you understand and troubleshoot. The insteon White pages will give you an understanding of insteon which would help in times of need. 

Quite a remarkable assertion for a problem that remains undiagnosed.

So when the ABS light comes on and the wheels start locking up on your Cadillac, do you insist you don't need an OBD-II tester with enhanced diagnostics, technical reference materials, and volt-ohm meters because you have a screwdriver, a hammer, and a flashlight? And then do you just buy another SUV when they don't do the job?

I've automated with Insteon devices for over 5 years and went to an ISY 16 months ago after the Hub died. I don't know anyone who has even seen one of these devices. Why would you think I am unfamiliar with the cookbook, the wiki, the forums (including at SmartHome), and other online sources? Just because I never had to post before, since other people had already provided insight into all the varied prior problems I've experienced?

I posted here only because I have not found another instance where simply re-learning the links restored operation, and I really hoped that someone else had already found and fixed the underlying cause.

 

[Thomas Roach]

7 minutes ago, larryllix said:

As a cheap and easy trial you could replace your SD card with a new one. After doing this I can say it is incredibly easy and takes about 30 minutes to be fully recovered. ISY eproms contain all the necessary boot up stuff.

Sent using Tapatalk
 

That's why I have two (hopefully quality) 32 Gb cards on the way. But I have not located recent documentation on the process, just the stuff for the ISY99i.

Can I log in through the normal network address to do this, or do I need to find something that uses the ISY's monitor port?

 

[Teken]

Just now, Thomas Roach said:

That's why I have two (hopefully quality) 32 Gb cards on the way. But I have not located recent documentation on the process, just the stuff for the ISY99i.

Can I log in through the normal network address to do this, or do I need to find something that uses the ISY's monitor port?

 

If you're going to bother replacing the Micro SD card I would encourage you to use a *High Endurance* card. Anything else is just a time bomb waiting to happen. Before you do anything insure you have two good backups of the system before proceeding. The details for the 99 Series Controller are the same steps to follow for the 994.

[Thomas Roach]

29 minutes ago, Teken said:

If you're going to bother replacing the Micro SD card I would encourage you to use a *High Endurance* card. Anything else is just a time bomb waiting to happen. Before you do anything insure you have two good backups of the system before proceeding. The details for the 99 Series Controller are the same steps to follow for the 994.

Really? The ISY uses the SD card that intensively? Makes it even more of a likely candidate.

I had ordered Samsung 32 Gb EVO Select cards, but I just added a Samsung 32Gb PRO Endurance card (I was out of spare cards, anyhow).

Assuming this works without a hitch, I'll run the current card through a testing utility after it's replaced and post the results.

 

[Teken]

Just now, Thomas Roach said:

Really? The ISY uses the SD card that intensively? Makes it even more of a likely candidate.

I had ordered Samsung 32 Gb EVO Select cards, but I just added a Samsung 32Gb PRO Endurance card (I was out of spare cards, anyhow).

Assuming this works without a hitch, I'll run the current card through a testing utility after it's replaced and post the results.

 

No, my comment is from the perspective of long term reliability. Any other card besides a *High Endurance* card will have a limited shelf life. What you sacrifice in terms of R/W speed you gain in durability. My reply on suggesting to use one is not reflective of the 994 Series Controller but more toward (IF) anything you rely or believe needs 99.9999 up time. It costs you very little to purchase the correct storage that is literally intended to be R/W XXX times and last when compared to other none High Endurance cards.

These cards are used everyday by professionals an avid Dash Cam users.

As the R/W almost always kills a standard Micro SD card . . .  

[lilyoyo1]

1 hour ago, Thomas Roach said:

Quite a remarkable assertion for a problem that remains undiagnosed.

So when the ABS light comes on and the wheels start locking up on your Cadillac, do you insist you don't need an OBD-II tester with enhanced diagnostics, technical reference materials, and volt-ohm meters because you have a screwdriver, a hammer, and a flashlight? And then do you just buy another SUV when they don't do the job?

I've automated with Insteon devices for over 5 years and went to an ISY 16 months ago after the Hub died. I don't know anyone who has even seen one of these devices. Why would you think I am unfamiliar with the cookbook, the wiki, the forums (including at SmartHome), and other online sources? Just because I never had to post before, since other people had already provided insight into all the varied prior problems I've experienced?

I posted here only because I have not found another instance where simply re-learning the links restored operation, and I really hoped that someone else had already found and fixed the underlying cause.

 

Wow! That's some Trump deflection at its finest. The reason I didn't think you did any of what you now act like you did is based off your original post (which I've added below for reference). Besides your own admission, restoring a device does exactly that. If there is something wrong with the memory of the isy or plm, restoring can accomplish exactly what it did.

On 3/1/2020 at 12:56 PM, Thomas Roach said:

 I don't know all the ins and outs of the system, and I really do not want to devote myself to in-depth study. I just wanted some help as I continue to grow old.

In regards to your auto comparison, what you were asking for was akin to having car troubles and wanting the source code from the mfg. to fix them. 

The reason why people are saying replace the SD card is that's the most likely culprit based on how everything works in the isy. The people who decided to help didn't need all the information you claimed you needed. The experience they've gained over the years as well as the willingness to take steps to learn about their devices gave them the knowledge needed to ascertain what is most likely the issue 

[Thomas Roach]

@lilyoyo1

I don't know all the ins and outs of the system. I don't fully understand the link structure and the communication protocols, I don't know how the ISY tracks devices, I don't know so much more about the varying devices and additional capabilities. There are people who do know all these things and so much more, people who should have been able to take the Event Viewer captures, look at the hex code, and at least give a hint about where to look. Did you glance at the Event Viewer logs I posted? Didn't the fact that I specified that it was a Level 3 Event Viewer capture give you a clue that perhaps I might have a little knowledge, since the default is a Level I view? 

As far as the SD card, I was the first to bring it up in this discussion. I didn't bother mentioning that I had already submitted a support ticket (unanswered to date, but it's not 48 hours yet) nor that I had already ordered cards once I made sure the ISY could now handle the 32 Gb instead of the 16 Gb limit in the documentation for the ISY99i

I've been looking for ideas and hoping that others could benefit from any solutions we discover, so that they won't have to try $100 PLMs, etc.

You've spent a lot of energy complaining that I should consult the wiki. Why not take some of that energy and spend it in updating the wiki itself?

 

[larryllix]

That's why I have two (hopefully quality) 32 Gb cards on the way. But I have not located recent documentation on the process, just the stuff for the ISY99i.

Can I log in through the normal network address to do this, or do I need to find something that uses the ISY's monitor port?

 

IIRC you just use puTTY to access SSH in the ISY.

You need the wiki instructions because although very simple, I dont remember the syntax to format the card inside ISY. The rest is standard admin console firmware, credentials, and restore your backup procedures.

 

Any SD card over 2GB is wasted. The old system cannot address more than ...

 

2^24 x 128 bytes/sector = 2GB

 

Of course the old cards may not have sector write scrambling or newer life extension techniques or speed either. I think 32 GB is as small as they make now.

 

Sent using Tapatalk

 

 

 

[Thomas Roach]

23 minutes ago, larryllix said:

IIRC you just use puTTY to access SSH in the ISY.

You need the wiki instructions because although very simple, I dont remember the syntax to format the card inside ISY. The rest is standard admin console firmware, credentials, and restore your backup procedures.

 

Any SD card over 2GB is wasted. The old system cannot address more than ...

 

2^24 x 128 bytes/sector = 2GB

 

Of course the old cards may not have sector write scrambling or newer life extension techniques or speed either. I think 32 GB is as small as they make now.

 

Sent using Tapatalk

 

 

 

Thanks for the heads-up.

Here's the wiki page I intend to use: https://wiki.universal-devices.com/index.php?title=ISY-99i/ISY-26_INSTEON:Replacing/Formatting_an_SD_Card

In the page, it mentions using Telnet (with a linked set of instructions) which I was able to enable as a Windows Feature (I'm running W10 Pro on the machine I use for the Admin Counsel). I managed to overlook that part after reading, "Please note: ISY supports up to a 16GB SD card with 4.2.18 firmware and above, any firmware before this only supports a 512MB SD card"

I started looking for 4 Gb cards (which is what came in the ISY), but it does seem like they're mostly obsolete. 32 Gb seemed to be the bottom limit for a quality affordable card, but the Samsung 32Gb PRO Endurance card was under $10.

 

[larryllix]

That's the one!

Sorry I cannot help much more with links etc.. as I am vacationing in Mexico right now with TapaTalk on a cell phone, and not many resources.

Sent using Tapatalk
 

[kclenden]

15 hours ago, Thomas Roach said:

There are people who do know all these things and so much more, people who should have been able to take the Event Viewer captures, look at the hex code, and at least give a hint about where to look.

I looked at your first Event Viewer log and added some comments in red.  Let me caveat my comments with the fact that all I know has been individually learned by reading any documentation I've been able to amass from google searches, and my experience with my own system.  The attached file is RTF format because my comments are in red text.  You should be able to open it with Wordpad or any word processor.  Everything below can be described, at best, as organized stream of consciousness.  Hopefully it will be useful, but YMMV.

Whether trying to understand my comments, or just troubleshoot your system, it's good to think of your system as three basic kinds of "modules": the ISY, the PLM, and devices:

• The ISY can only communicate with the PLM (via the serial cable between the two)
• The PLM can communicate with either the ISY (via the serial cable between the two) or devices (via the powerline)
• Devices can communicate with the PLM (via the powerline or RF) or other devices (via the powerline or RF)

Other than when linking, the PLM and Insteon devices can only communicate with other devices that are known to them.  It is the linking process that introduces the PLM and devices to each other.  When they're introduced (via linking), the PLM and devices are presented as either a Controller or a Responder in a group. When you create a Scene using the Admin Console, behind the scenes (all puns intended) the ISY is creating an Insteon group and linking each device as either a controller or a responder.

Linking is so central to how Insteon works that all three "modules" maintain Link Tables:

• Each device contains a row in its table for every group it belongs to with data that indicates the group and whether the other device (by address) is a controller or a responder
• The PLM contains a row in its table for every group it belongs to with data that indicates the group and whether the other device is a controller or a responder
• The ISY maintains a copy of the link table in every device as well as a copy of the link table in the PLM.  It doesn't use these to communicate with either the PLM or the devices, but rather as a backup.

With the ISY as your Insteon Hub, it manages all of the linking (i.e. introductions).  It introduces the PLM to each device since it has to use the PLM to communicate with devices.  When it first links the PLM to a device here's what happens:

1. A row (link) is put in the device's link table with the PLM's address and an A2 which indicates the PLM is a responder to that device
2. A row (link) is put in the PLM's link table with the device's address and an E2 which indicates the device is a controller of the PLM
   1. These two rows usually represent group 0 for that device
3. A row (link) is put in the device's link table with the PLM's address and an E2 which indicates the PLM is a controller of that device
4. A row (link) is put in the PLM's link table with the device's address and an A2 which indicates the devices is a responder to the PLM
   1. These two rows usually represent group 1 for that device

Those four steps are done every time you link an Insteon device to your ISY.  This allows the ISY to control the device and it allows the ISY to receive state changes from the device (without asking).  It means that if you list your PLM Link Table, there should be at least two times as many rows in the link table as devices.  In actuality it's usually even more because some devices (like Keypads) count as more than one device since they have multiple nodes (e.g. buttons).  In addition, whenever you create a Scene using the ISY, you're really creating an Insteon group and every device (including the PLM) in the group will have rows in its link table indicating the group membership.

Whenever a message is sent out on the Insteon network, every device on the network (including the PLM) looks at the message to determine if it's meant for it, and if so whether it's coming from somebody it knows.  It does this by checking its Link Table.  The Link Table has a row for every controller it knows (as well as the group it controls) and a row for every device that it can control (as well as the group it responds in), so it's a simple matter for a device to determine whether the message is intended for it.  If it determines that the message isn't for it, it doesn't take any action other than to repeat the command (after decrementing the Hops Left count).

Whew!  I hope you stayed with me because with that background you can begin to troubleshoot any problem.  You start by defining what the problem is, and then listing what could cause the problem.

• The ISY suddenly stopped controlling all Insteon switches and relays, but it would update the status at the console.  Commands produced "Cannot Communicate with XXXXX". X-10 devices still worked.
  1. The ISY might not be communicating with the PLM
  2. The PLM might not be communicating with ALL devices
  3. ALL devices might not be receiving PLM communication
  4. ALL devices might not be accepting commands from the PLM

The easiest way to rule out each of the items would have been to look at an Event Viewer Level 3 log.  But we don't have one of those from when you were originally having problems.  We can likely rule out #1 because your later Event Viewer log shows no problem with communication between the ISY and PLM.  We can likely rule out #3 also because your later Event Viewer log shows no communication problems AND because you say that X-10 continued to function which means the powerline was at least able to transmit X-10 signals and Insteon signals are designed to be more reliable.  We're left with #2 and #4 to consider.  For the PLM to stop communicating with ALL devices, all that would be needed is for its Link Table to become corrupt.  For all devices to quit accepting commands from the PLM, the Link Table in every device would have to become corrupt.  One device table (i.e. the PLM's) becoming corrupt seems a lot more likely than 30.

The problem with the corrupt PLM Link Table theory is that you were still able to see status updates in the console.  That means that the PLM Link Table must still have had rows indicating it was a responder to each device.  It seems highly unlikely that a PLM Link Table would become corrupt in such a way that all of the PLM controller records got wiped out but all of the PLM responder records remained.  Another problem with the PLM Link Table theory is that nothing got better when you performed a "Restore PLM".  That should have rewritten all the links from the ISY's copy of the PLM Link Table into the PLM, and something should have gotten better.  Likewise when you did "Replace PLM", all the links from the ISY's copy of the PLM Link Table should have been written to the new PLM, and every individual copy of a device's Link Table should have been updated with the new PLM address.  But that didn't fix things either, which would seem to indicate that the ISY's copy of either or both the PLM Link Table and/or each individual device Link Table was corrupt.  In all honesty, with all of the things that you have done to the system between the first problem and now, I don't think it will be possible to determine what happened.

Looking at the Event Viewer log that you provided, a couple things are clear:

1. The log shows the ISY and PLM communicating with each other just fine
2. It shows the PLM and devices communicating with each other just fine
3. It shows two devices rejecting a command from the PLM to turn On
4. It shows two devices accepting a command from the PLM to send their light status
5. It shows the PLM receiving a group broadcast message from the mini-keypad telling group 6 to switch to full On
6. It shows the PLM entering ALL-Linking mode so that it will be a controller for any device that responds
7. It shows device 33.1C.87 responding
8. It shows the PLM exiting ALL-Linking mode
9. It shows the PLM sending a command to 33.1C.87 telling it to switch to full On
10. It shows 33.1C.87 responding that it accepted the command and has switched full On
11. It shows the PLM sending a command to 33.1C.87 to go to full Off
12. It shows 33.1C.87 responding that it accepted the command and has gone to full Off

Some comments:

• #3 above would happen if the two devices had corrupt Link Tables.  They recognize their address in the message, but aren't able to find a record in their Link Table that says the PLM is a valid controller for the specified group.
• #4 probably works because the devices are able to find a record in the in their Link Table that says the PLM is a valid responder, and a Status Request command is really just a way for a device to a request status update rather than wait for the device, acting as a controller, to send a status.
• #5 shows that the PLM link table still contained a record indicating the PLM was a responder to the mini-keypad, and if the lights came on, it shows that they still had records in their Link Tables telling them that the mini-responder was a valid controller.
• #7 would create a row in 33.1C.87 that restored the PLM as a valid controller of 33.1C.87
• #10 and #12 shows that 33.1C.87 did actually create a row in its Link Table establishing the PLM as a valid controller.

It would be interesting to see the Link Tables for your PLM as well as devices 33.1C.87, 3E.33.FE, 44.8C.E7 and one or two other devices.  You can use Tools->Diagnostics->Show PLM Links Table and Tools->Diagnostics-Show Device Links Table and click "Save".  That will save each in an XML format.  The reason they would be interesting is because you say, "I tried adding new devices under Link Management, ignoring the "already added" message and keeping all the established links."  That is a recipe for a very confusing device Links Table, and I'm interested in what the result was.  My guess is that you have some of the original corrupt links, then some links from when you did the "Replace PLM" and then some links from when you added the devices in again.  I really don't know what a device would do if all of those links were in its Links Table.  Generally when you link a device to to the PLM, you don't want to keep existing links because you don't know what those existing link may be.  I understand why you did it that way this time, but it will more than likely come back and bit you later.

 

 

ISY-Events-Log.v5.0.16C__Sun 2020.03.01 04.35.45 PM.rtf

[Thomas Roach]

@kclenden

Thanks! Your comments and explanations are quite informative.

In hindsight, I wish I had copied more exchanges and paid more attention to details such as the hex device address at the top of the screen when the problem was occurring (it has not resurfaced since I cleared the logs, which were about 10 Mb). But I posted the Event Viewer listings without being able to interpret them at the time.

I also now realize that the ISY was not reporting any of its communication regarding the issue WHEN it was occurring. So let me add some additional information about the first few lines. I had already established that trying to command on the Porch Lights (33.1C.87) set the red exclamation mark and the ISY would not update status, but after hitting Query the green "Write" symbol replaced the red exclamation mark and the status would update when I activated the switch manually. The lines prior to this point seem to be the Event Viewer starting up:

I hit the Query button on the Porch Lights page (should have gone to 33.1C.87)

Sun 03/01/2020 04:18:01 PM : [  3E 33 FE 1]      ERR   0

I hit the On button on the page

Sun 03/01/2020 04:18:06 PM : [  3E 33 FE 1]      ERR   1

I hit the Query button on the page

Sun 03/01/2020 04:18:19 PM : [  3E 33 FE 1]      ERR   0

Sun 03/01/2020 04:18:36 PM : [All         ] Writing 0 bytes to devices

Sun 03/01/2020 04:18:36 PM : [All         ] Writing 0 bytes to devices

...

And so on until I hit the keypad (the button turns on 33.1C.87 and 3E.33.FE and may still contain a link to a long-offline defective Insteon switch, since the keypad probably predates the ISY). Then this message exchange started:

Sun 03/01/2020 04:25:33 PM : [INST-TX-I1  ] 02 62 3E 33 FE 0F 11 FF
and etc.

My hunch is that the ISY went into an error mode before communicating with the PLM.

 

[kclenden]

8 hours ago, Thomas Roach said:

I hit the Query button on the Porch Lights page (should have gone to 33.1C.87)

Sun 03/01/2020 04:18:01 PM : [  3E 33 FE 1]      ERR   0

While I can't setup my system to mirror the state of your system when you were having problems, I have tried a couple things to get close.  I have an Outdoor Module that I only have plugged in for Christmas Season.  When I unplug it at the end of the season, my ISY will give me a red exclamation point and tell me that it can't communicate with the device.  So I simply "Disable" the device until the next Christmas season.

Using that "missing" Outdoor Module, I was able to create an Event Viewer log similar to yours.  First I "enabled" the missing device and then shutdown my Admin Console.  I started the AC up again and started the Event Viewer but left it on Level 1.  Then I "disabled" the device and saw an ERR 0 in the Event Viewer.  So I tried to "enable" the device and after several seconds got a "Unable to communicate with device" pop-up message.  An ERR 1 appeared in the Event Viewer.  So I "disabled" the device again and an ERR 0 appeared in the Event Viewer.  So I "enabled" the device again and received an ERR 1.  Next I tried to turn the device "On".  Nothing appeared in the Event Viewer and no pop-ups appeared on my screen.  Then I tried to "query" the device.  I got a series of "System Busy" messages that popped-up and then disappeared.  Nothing appeared in the Event Viewer.

Next I tried everything above again, only with the Event Viewer set to Level 3.  The results were exactly the same, except:

• after each attempt to "enable" the device, three combos lines of "02 62 xx xx xx xx 19 xx" and "02 62 xx xx xx xx 19 xx 06" appeared in the Event Viewer before the ERR 1
• after each attempt to "disable" the device, a "[D2D Event    ] Event [xx xx xx x]" appeared in the Event Viewer before the ERR 0
• after each attempt to turn ON the device, two combo lines of "02 62 xx xx xx xx 11 xx" and "02 62 xx xx xx xx 11 xx 06" appeared in the Event Viewer without a following ERR 1
• after each attempt to QUERY the device, three combos lines of "02 62 xx xx xx xx 19 xx" and "02 62 xx xx xx xx 19 xx 06" appeared in the Event Viewer without a following ERR 1

Based on all that, is it possible that the first seventeen lines of your Event Viewer log were actually at Level 1 and that the ERR 0 and ERR 1 message were actually a result of you trying to "enable" and "disable" the device?  And then starting at line 18 the Event Viewer was change to Level 3 and you then started trying turn on the device and query it?

I'd still be interested in seeing some of your Link Tables.

[Thomas Roach]

13 hours ago, kclenden said:

Based on all that, is it possible that the first seventeen lines of your Event Viewer log were actually at Level 1 and that the ERR 0 and ERR 1 message were actually a result of you trying to "enable" and "disable" the device?  And then starting at line 18 the Event Viewer was change to Level 3 and you then started trying turn on the device and query it?

Partially right. It seems that the first lines actually cover the start-up of the Admin Counsel, not the Event Viewer. I thought the Event Viewer was cleared after I had saved the initial Level I/Level 3 log at 04:33:46 PM, but apparently that's not the case. Frankly, all I knew about the Level 3 Event Viewer at the time was that  @Michel Kohanim thought it was helpful in diagnosing similar problems. Other than that, it just looked like a bunch of abbreviations and hex codes.

However, while I was not enabling and disabling the devices during the session, I very well could have been trying to turn on 3E.33.FE during that time. In that case, my hypothesis of "wrong device addressing" seems invalid.

Attached are the PLM Links Table, the links tables of 3E.33.FE and 33.1C.87, and the Topology. Let me know if that helps.

So I take it the PLM is not a "dumb modem" but maintains its own database of links, etc., that could become corrupted (perhaps by the ISY going offline while communicating with the PLM)? When does the ISY build the file it uses to restore the PLM? Did I just "restore the PLM" (both old and new) with corrupted information garnered after the fact?

I'm left with these circumstances: 1) the problem started after I power-cycled the ISY while working on my network; 2) the problem reoccurred in varying degrees until I cleared the logs, which were about 10 Mb in size; 3) just "re-learning" the links resolved the problem, at least temporarily; and 4) the new SD card has arrived and the new PLM has joined the pile of spare Insteon devices.

 

PLM Links Table.v5.0.16C__Sat 2020.03.07 09.20.50 AM.xml Device Links Table.v5.0.16C__Sat 2020.03.07 09.21.49 AM.xml Device Links Table.v5.0.16C__Sat 2020.03.07 09.22.40 AM.xml ISY Topology.v5.0.16C__Sun 2020.03.01 11.58.28 AM.html