How can I diagnose ISY hardware failure?

[bradshawkyle]

I have a 994iPro running 5.3.0 that's been in service about 7 1/2 years, never had a problem with the ISY side of things. I've replaced a few PLMs, but otherwise it's proven extremely reliable.  Recently, the system has been acting in what I can only describe as "inconsistent" then it comes to running programs.  For example, I have a program that texts me 9am every weekday if I forgot to arm the alarm.  Two weeks ago Tuesday it sent at 9:54am, then ran perfectly at 9am until the following Tuesday when it sent again at 9:54.  Then back to the usual time until yesterday when it sent at 1:26pm and today at 4:12pm.

Another example is a program I have to turn on the porch light 30 minutes before dusk....some days it runs and others it doesn't, and using the ISY to control the device directly works fine.  If the problem were relates to controlling Instron devices I'd suspect the PLM had failed, but given the text message issues I'm suspecting it's related to running programs in the ISY.  I checked the programs and nothing has changed in months - should I just replace the ISY or is there something I can do to troubleshoot?

[stillwater]

The logs should give you some idea.  

It's possible the power supply (wall wart) is dying and the ISY is rebooting after some time, in which case replace the wall wart.   Power supplies are the most common source of failure in electronics in my experience.   Other than that I can't figure out what might be going wrong.  

[Michel Kohanim]

@bradshawkyle, in addition to excellent suggestions by @stillwater, when this happens next, please check the Last Run Time column for your program. If the time is accurate, then the problem is signal (for INSTEON) and ISP/carrier/SMTP server (for notifications). If the times are not correct either, then either the clock in 994 is dying or you have other conditions in the program the impact the time.

With kind regards,
Michel

[bradshawkyle]

Fantastic suggestions, thanks to you both.  The good news is it's 8:22am here, so the first opportunity to diagnose is in 38 minutes. ?

[bradshawkyle]

And while we wait, it's worth noting the "alarm" program I created (which sends notifications when motion is detected) is dutifully sending notifications via text despite the program being in "disarmed" state for the past 20 minutes.

[bradshawkyle]

90 minutes past the program run time, still no text notification.  The Last Run Time, however shows it ran at 9am, which would indicate I have problems with both signal and notifications which started at the same time. Tests using SP/carrier/SMTP server turn immediate responses as expected with no time delay.  Signal tests where I control Insteon devices directly also perform perfectly with no delay.  I'll follow @stillwater 's suggestion and order a power supply.  If that doesn't work, I'm willing to replace the PLM and rebuild my notifications, but is there anything else I should test or examine before committing to that?

[stillwater]

Sounds like your ISY is on the whole time so I doubt it's an ISY power issue. 

 If you  test (manually run)  the network resource that does the notification does that go through immediately?  Maybe add a line to the program that increments  a  variable so you can confirm that the whole program executed?  If you look at the ISY logs do they show appropriate device activation and communication? 

[bradshawkyle]

Quote

If you  test (manually run)  the network resource that does the notification does that go through immediately?

Yup, text and email notifications.

Quote

Maybe add a line to the program that increments  a  variable so you can confirm that the whole program executed? 

Unsure how to do that, but I'll do a little research and see if I can add it.

Quote

If you look at the ISY logs do they show appropriate device activation and communication? 

I looked at device activations related to the programs mentioned above and the logs show the command was sent at the requested time.

[stillwater]

Based on the communication being sent it sounds like the program executed to that point, and so I doubt changing a variable will provide additional diagnostic utility.    I think this is beyond my experience/expertise.   It's hard to understand both why the failures of the two modes of communication started happening at the same time and why they are continuing.   The Insteon issue could be a PLM failure but that doesn't explain the seeming failure of the network resource to execute properly.  

[stillwater]

Clutching at straws here --  before replacing the ISY I'd try the following

1) I assume you've tried rebooting the ISY?  If not that's an easy thing to try

2) Maybe a new SD card?  ( @Michel Kohanim  would have a better idea whether this is a conceivable fix for the problem) 

 

[bradshawkyle]

Quote

 I assume you've tried rebooting the ISY?  If not that's an easy thing to try

Oh yeah, it's my first go-to when things are acting up.

Quote

Maybe a new SD card?

I don't think my ISY has an SD card, but will double check.

 

Now that the porch light program ran and failed (light was not turned on) again, I noticed the program ran at 4:40pm yet the log shows the command was sent to the device at 4:58pm.  I'm not sure of this is helpful information, but thought I'd pass it along just in case.

[bradshawkyle]

Update: The system has become self-aware and is attempting to assert its authority by barraging our cell phones with text messages from events logged 14 hours ago.  14 and counting from a single event.  I disabled the program, but no luck.  I have now deleted all notification settings (containing text and email addresses) in the hopes of making it stop.  If that fails, I'll unplug the ISY and report back in the morning.  If nothing else, I hope you guys are getting a chuckle out of my minor inconvenience. ?

[larryllix]

On 2/3/2022 at 1:21 AM, bradshawkyle said:

Update: The system has become self-aware and is attempting to assert its authority by barraging our cell phones with text messages from events logged 14 hours ago.  14 and counting from a single event.  I disabled the program, but no luck.  I have now deleted all notification settings (containing text and email addresses) in the hopes of making it stop.  If that fails, I'll unplug the ISY and report back in the morning.  If nothing else, I hope you guys are getting a chuckle out of my minor inconvenience. 

I find the same thing with many other items also. it is not just ISY trying to take over.

With the pandemic congestion, I get emails and text messages warning me of events that happened the previous day.
This pandemic is slowing things down more than we want to admit.

Also with truckers blockading major routes it may be dragging the data speeds down with it. I know the truckers all have CB radios and they use RF frequencies that may be killing our WiFi items. We need to blame them possibly.

Sent from my SM-G781W using Tapatalk

 

[bradshawkyle]

The texts rolled in last night in a few batches even after the ISY was disconnected, which is pretty interesting. They would come in groups of three, same alert, time stamp was between 7 and 9am and never the same time on any alert.  Truly bizarre behavior.

Regarding @larryllix's comments on data congestion - we're fortunate to be relatively unaffected here in Washington State.  I've been working from home the past two years now and monitor ISP and network health daily, so as far as our "stuff" goes everything is fine. Of course there's no telling how Verizon is doing, but I can tell you we haven't experienced any other delayed text messages, including ISY tests.

The new power supply arrived and is installed, will monitor the aforementioned porch light program and see if it works.  For now I am leaving all notification data (for sending text and emails) deleted to see what happens.

[MrBill]

If your notifications are email to text messages, that is likely the issue.    The notifications are actually going out at the correct time, but they get hung up in the email to text gateway at your carrier.  You can thank spammers that are using the gateway for spam for creating the problem.

there are several solutions, the best of which is using pushover.net with a nodeserver.  If you have Polisy I'd suggest @Jimbo's nodeserver called Notification, If you don't have a Polisy but do have a paid Portal account there is a cloud nodeserver called Push that will also work with pushover.net.  Essentially to use pushover you buy the app once for as I recall 4.99 and there is no monthly charge.   Pushover notifications are instant. 

[bradshawkyle]

Update: deleting all notification configurations resolved the issue immediately.  Before deleting, I tested the text notifications and they worked instantly, with no delay.  I went ahead and deleted them and ran with none for 2 weeks, then email only until this weekend.  Emails worked fine.  I added text notifications back in, tested fine, but today the same issues came back: receiving notification texts from events occurring at 8am multiple times and up to 12 hours late.  I deleted all text notification data about an hour ago and just received a text from this morning's notification.  They'll stop eventually.  

I have decided to ditch the text message notifications entirely, use the (character limited) push notification feature in Mobilnc to alert me, then send the details in email.  

BTW: the issue I was having with the porch light program was caused by a bad LED bulb, which I learned AFTER replacing the dimmer and encountering the same issue. ?  The bulb worked intermittently and when it acted up it completely freaked out the dimmer and caused it to stop working.  New bulb = problem fixed, AND I didn't toss the old dimmer so I'm not kicking myself.