Intermittent Z-Wave disappearance

[steve-elves]

I'm pretty much at my wit's end, and the usually-stellar support folks at UD are running out of ideas as well.  They suggested that I run it by the equally-stellar folks in this here forum.

My home system has an EISY at its core, talking to 45 Z-Wave devices (holding 219 values or "nodes" according to the ST Inventory plugin), 20 Node Server nodes, 27 Insteon devices, and one Brultech GEM.  The system contains 125 programs that use 40 State variables and 45 Integer variables.  I use UD Mobile to access the system locally and remotely, in addition to program access through my Windows 11 PC.  Both the EISY and HA boxes (as well as my router and switches) are powered through relatively-new battery backup UPS's.  I haven't added any new devices to the system for a couple of months.

I have configured Home Assistant in a mini-PC with the UD Integration; I am passing information from 96 devices in the EISY to my HA for display purposes only.  At this point, I haven't configured any automations in HA that read or write to any device in the EISY.

Everything worked more or less flawlessly until mid-February this year, which is when I installed the latest update to IoX (v.5.9.1).  That, too, seemed to work just fine, for the first 10 days or so, and Michel assures me that the update is almost certainly NOT at fault.  Suddenly, my heating programs (which turn Z-Wave and NuHeat WiFi thermostats down and turns the Z-Wave heavy-duty relay that feeds power to my electric water heater off in the evening, and turns everything up and on in the morning) was failing to write the appropriate setpoints.  I verified that the programs had indeed actually run on time; the commands were simply either being ignored or not sent.

When I looked at the various devices in the Admin Console or on UD Mobile, the data fields were blank and I couldn't send new setpoints or query the devices.  I could go to each physical device and manipulate it locally, but they were not responsive to commands from the EISY.  

I tried several different methods to bring things back to proper communication.  Several reboots, restoring of a recent (but post IoX upgrade) backup, "healing" the Z-Wave network multiple times, changing the comm speed of the Z-Wave network from 10 to 15 (default) to 20 and back to 15, disconnecting the HA box entirely, etc. I made sure that any Z-Wave devices that appeared in the configuration in the Admin console but that were disconnected in the field (such as some Christmas light controllers) were disabled in Admin.  I removed two or three obsolete devices that were configured but were actually nonexistent.  

None of these fixes have been permanent.  The last time I did a network heal was on the 15th or 16th of March, and the system worked perfectly for about 6 days - then, last Saturday, the "heat up" programs in the morning ran but the thermostats and hot water tank remained low/off.  Having only lukewarm water for my wife's morning shower is NOT a great way to start a day!

UD has looked at my Event Logs and Error Logs and all the Z-Wave logs, and couldn't turn up any obvious answers.  I'm hoping one or more of you people can give me some ideas as to how to further troubleshoot the issue!  Thanks in advance.

[Guy Lavoie]

How long have you had the "relatively new" UPS's? Just one of those things that can create electrical noise issues. Have you tried eliminating them, for test purposes? If your problem occurs frequently enough, you should have a quick enough conclusion as to it they're a problem or not.

[Techman]

I don't think the UPS's could be the problem as Zwave is an RF only platform. UPS's are  known to create powerline noise which impacts Insteon devices but not Zwave. 

Have any RF devices been added to the home recently. Another possibility is that there's a defective Zwave device in the mesh network that's causing interference.

You can also try adding a zwave repeater/amplifier to your system.

Zooz 800 Series Z-Wave Plus Range Extender ZAC38 - The Smartest House

[Guy Lavoie]

8 hours ago, steve-elves said:

Suddenly, my heating programs (which turn Z-Wave and NuHeat WiFi thermostats down and turns the Z-Wave heavy-duty relay that feeds power to my electric water heater off in the evening, and turns everything up and on in the morning) was failing to write the appropriate setpoints.

Rereading your post caught my eye at the above quote. Both Z-Wave and wifi thermostats aren't getting updated? Now that would appear to be more of a program execution issue. The fact that it started after an upgrade to 5.9.1 is also suspect. I'd add test code, such as incrementing a variable in the same program, just to see that it's going through the statements. One before and one after. Something like this:

Then increment variable 1

Then set Z-wave termistat

Then set wifi thermostat

Then increment variable 2

And check it every day to keep track of the variable values (or write them down along with the date).

[steve-elves]

3 hours ago, Guy Lavoie said:

How long have you had the "relatively new" UPS's? Just one of those things that can create electrical noise issues. Have you tried eliminating them, for test purposes? If your problem occurs frequently enough, you should have a quick enough conclusion as to it they're a problem or not.

Thanks for the input.  I more-or-less ruled out the UPS as an issue simply because the problem only occurs with Z-Wave devices - not WiFi or Insteon ones.  That led me to conclude that there was a communication issue rather than noise.  You have reminded me that noise can be insidious in its effects and I shouldn't rule it out offhand.  I will take your suggestion and do the experiment if I can't find another solution - I don't want to remove my controller from the UPS unless I have to, as we are in a rural area that is subject to power outages and sags.

[steve-elves]

1 hour ago, Techman said:

I don't think the UPS's could be the problem as Zwave is an RF only platform. UPS's are  known to create powerline noise which impacts Insteon devices but not Zwave. 

Have any RF devices been added to the home recently. Another possibility is that there's a defective Zwave device in the mesh network that's causing interference.

You can also try adding a zwave repeater/amplifier to your system.

Zooz 800 Series Z-Wave Plus Range Extender ZAC38 - The Smartest House

Thanks for the input.  I, too, had more-or-less discarded the idea that UPS's could be the fault, but one should never be too quick to eliminate possible solutions when the problem isn't obvious.

I think I agree that faulty or weakening Z-Wave device(s) may be the issue.  Some of my Z-Wave units are getting a bit old now, and (just like me) they likely aren't functioning as well as they used to.  The layout in my house is sub-optimal for good RF communication, and my EISY with Z-Wave dongle is in the basement alongside my PC.  I've got a couple of older Z-Wave repeaters, but I long suspected that they aren't really doing much.  I think I will go ahead and order a couple of the Zoos range extenders.

[steve-elves]

57 minutes ago, Guy Lavoie said:

Rereading your post caught my eye at the above quote. Both Z-Wave and wifi thermostats aren't getting updated? Now that would appear to be more of a program execution issue. The fact that it started after an upgrade to 5.9.1 is also suspect. I'd add test code, such as incrementing a variable in the same program, just to see that it's going through the statements. One before and one after. Something like this:

Then increment variable 1

Then set Z-wave termistat

Then set wifi thermostat

Then increment variable 2

And check it every day to keep track of the variable values (or write them down along with the date).

Thanks, Guy!  I should have been clearer in my explanation.  I have about 6 WiFi thermostats (NuHeat brand) plus 4 Z-Wave ones.  The WiFi thermostats are updated properly - it's only my Z-Wave thermostats, load switches, and lights that are intermittently functional.  I like your idea for using variables to check the operation of individual program commands.

[Guy Lavoie]

If you have spare Z-wave plug in modules, have you tried adding a few, to try and re-jig the network?

[steve-elves]

5 minutes ago, Guy Lavoie said:

If you have spare Z-wave plug in modules, have you tried adding a few, to try and re-jig the network?

I don't, but I have a couple that I might be able to move.  Thanks for the idea!

[Techman]

@steve-elves

Are the Zwave devices you're having issues with 300, 500, or 700 series?

Being that Michel has eliminated possible eisy issues, I would focus on marginal communications between the eisy and your devices. i.e. a zwave device that's noisy or the need for a repeater or two

[steve-elves]

13 minutes ago, Techman said:

@steve-elves

Are the Zwave devices you're having issues with 300, 500, or 700 series?

Being that Michel has eliminated possible eisy issues, I would focus on marginal communications between the eisy and your devices. i.e. a zwave device that's noisy or the need for a repeater or two

I've had Z-Wave devices for 7 or 8 years, so no doubt there are some from each series - or, st least, 500 & 700 for sure. I suspect that you are absolutely correct in your analysis, if for no other reason than the random nature of the problem.  I've ordered a couple range extenders, and hopefully they will solve my issue!

[Techman]

After you install the extenders let me know if they solved your problem

The zooz 800 serie extenders have a battery backup. If your eisy router and modem are on a ups you can set up notifications to alert you of power failures. 

[lilyoyo1]

6 hours ago, steve-elves said:

Thanks, Guy!  I should have been clearer in my explanation.  I have about 6 WiFi thermostats (NuHeat brand) plus 4 Z-Wave ones.  The WiFi thermostats are updated properly - it's only my Z-Wave thermostats, load switches, and lights that are intermittently functional.  I like your idea for using variables to check the operation of individual program commands.

With 10 thermostats, I assume you have a fairly large home. Are your zwave devices grouped together or spread out? Is your eisy centrally located or on one end trying to communicate to another? 

Are your issues with every zwave device or specific ones? If it's specific, that's an easier fix. While adding repeaters can work, you want to do it strategically to get the most out of them.