automatewhatican

Interesting. Another lock-up occurred overnight. I haven't rebooted yet. I need an escalation plan; the repeated hard fails are not sustainable. Does anyone have a suggestion? I will pull a ticket, referencing this thread. The display tail of error messages this morning on console seems the same last two times:

It's in a computer closet with a reasonably controlled environment. Current ambient is only 81.4F. eisy itself feels warm to the touch, not hot.

On plugins for this eisy, I'm running production versions of OpenWeatherMap 4.0.3 and iAquaLink 3.1.10 plugins and have a production version of Notification 3.6.32 that is UNconnected in PG3. On the ISY side, I have a pretty good set of Z-Wave and Insteon nodes, probably at least 20 of each, although I can do an inventory if helpful. Zwave is mostly AeoTec multisensors and switches and a home energy meter node that has been unpaired for a couple years but for which an inactive node remains. I also have three Honeywell T6 thermostats and a Yale lock. I have about 20 programs; nothing super complicated, but a fair amount of environmental monitoring, and motion or time-cued lighting and thermostat setpoint controls. Insteon is entirely lighting controls. Other than the standard 3am Query and any usual eisy/freeBSD housekeeping, I don't think there is anything particularly noteworthy in the overnight execution. Z-Wave is an original ZMatter; I've never updated ZMatter firmware. Other than that, packages are all pretty current. Does anything catch your eye as a good line of inquiry/investigation?

Overnight, memory utilization (as indicated in eisy-ui) spiked to 94%. Less than 0.5 GB remains free per system monitor (189M Active, 700M Inact, 7812K Laundry, 6270M Wired, 465M Free). Nothing strikes me as inherently problematic in that, but it might be notable that someone at UD decided to code the thermometer bar to present in RED rather than BLUE at these levels. Is there something else I should be checking? ... or just wait for lock up and then pull some log?

Will do, Guy. So far so good; the memory utilization (as indicated in eisy-ui) has been hovering around 25% and about 6GB of free memory is showing in freeBSD system monitor. Agree on unusual. Reliable, non-stop unattended performance is really the isy value proposition, but these recurring lockups are now of concern. I will give it 48 hrs, checking memory; failures appear to occur in early AM. If there is anything else you'd suggest monitoring or checking, let me know.

Thanks, Guy. I understand the up pressure from memory paging and caching. The real problem is the repeated lock ups, requiring hard restart. The eisy-ui's RED symbology for the 95% utilization is what caught my attention. Next time, I'll check the proportion of the pool containing active pages, leaving aside dirty inactive pages.

Done. As expected, down to 24% (1.8 GB / 7.7 GB). I guess we'll have to see if it creeps back up over the next 36 hrs.

I've recently had two lockups of the same eisy since updating to 14.4-RELEASE-p6. The lockups were roughly 48 hours apart. In each case, the console showed a controller timeout and register dump together with an indication of an unrecoverable I/O failure in the ZFS pool. See below: After reboot, I ran a zpool status, which reported no errors. It's now been about 36 hours since the last lock up, and I noted the following in eisy-ui: Is there possibly a memory leak in IoX 6.1.2_7? What do you suggest?

Thanks for the update, Javi. And thanks for the thorough problem summary. Is there a way to provision a portal/proxy connection on an ISY944 running 5.3.4, but which is not current connected through the proxy? For some reason, I do not get an opportunity to approve a portal connection for the affected uuid. I also do not get the Configuration | Portals tab in Admin Console. If we could get communications with the ISY944 established through the portal/proxy that would be a useful workaround until the remote device can be physically upgraded to (replaced with) an EISY.

I have this same issue. I have been able to resolve for an EISY deployment, but I have not been able to resolve using Javi's workaround for an ISY. For the ISY (running 5.3.4), no Portal tab ever presents, so the portal communications based workaround is unavailable. Help please.

Thanks dbuss. Were you seeing successful reception of Zwave message frames, but failure to transmit?

I don't think it's a USB issue as the failure has presented w/ and w/o a short (3ft) intervening USB cable AND when the dongle is plugged directly into various of the USB ports of the EISY. I'll pull a ticket.

Since transitioning from ISY944/500series ZWave to EISY/ZMatter, I have had a problem with ZWave communication failures that affect ZWave communication with my entire ZWave network (20-30 devices) is no longer controllable. The failure seems to present after a week or two of uptime and I've been "fixing" by rebooting and removing power cycling various components (including the EISY and ZMatter). Rather than power cycling, today I went looking for clues. I noted the following based on event logs and behaviors: outbound communications from the EISY/ZMatter show failure (in logs). This is consistent with the observable that device states do NOT change based on commands initiated at UDMobile or admin console. inbound communications do seem to be being received, even when the ability to control ZWave devices is down, and mirrored states are generally being updated based on states changed at ZWave devices. during normal operation, I would expect periodic and somewhat frequent (once a minute) telemetry from an Aeotec ZW095 Home Energy Meter. However, during the ZWave down condition, I'm not receiving those. Today, rebooting EISY and removing/restoring power to the ZW095 seemed to resolve the problem. For years my ISY944 was rock solid with the network of the same ZWave devices. This period human-in-the-loop reboot that I'm now experiencing is not going to work. For a while, I just assumed that a forthcoming IoX release would fix; however, the failures remain after updating to 5.8.3. Any ideas? How do I open a ticket? ... and what other data should I try to collect?

Yes, on the z-wave alliance site, "Z-Wave Plus Version 2" signifies a series 700 device.

Resolved with Pro upgrade. I concur with @MrBill, non-Pro node limit is easy to hit, particularly with some of the newer Z-Wave devices that result in many subsidiary ISY nodes. Based on a quick manual count, I’m carrying 30 physical devices that are Z-Wave and 22 that are Insteon (including 5 KPLs).

Thanks. Done both. Same problem. Went on to delete some extraneous device nodes and got further on next add node try, but this time got a series of "too many nodes" messages after first three nodes added. Seems like the node count tallies more quickly than I would have thought. Maybe all the subsidiary nodes of newer Z-wave devices? Will try a PRO upgrade next.

I've been trying (and failing) to add a new Z-Wave device to the ISY. I've gone through multiple remove/add cycles but get the same result as illustrated below: Sun 12/26/2021 10:51:52 AM : Listening for Z-Wave devices to remove from the network Sun 12/26/2021 10:51:55 AM : Removing Z-Wave devices stopped Sun 12/26/2021 10:51:56 AM : Device 79 excluded from Z-Wave network Sun 12/26/2021 10:52:07 AM : Listening for Z-Wave devices to add to the network Sun 12/26/2021 10:52:15 AM : Z-Wave device detected, retrieving info 2 Sun 12/26/2021 10:52:15 AM : Z-Wave device detected, retrieving info 3 Sun 12/26/2021 10:52:22 AM : Z-Wave device detected, retrieving info 5 Sun 12/26/2021 10:52:23 AM : Adding Z-Wave devices stopped Sun 12/26/2021 10:52:24 AM : Z-Wave device detected, retrieving info 6 Sun 12/26/2021 10:52:24 AM : Device 80 of type 4.16.6 included into Z-Wave network Sun 12/26/2021 10:52:30 AM : Add Node failed: Cannot create primary node for new device Sun 12/26/2021 10:52:31 AM : Add Node failed: Cannot create primary node for new device Sun 12/26/2021 10:52:31 AM : Add Node failed: Cannot create primary node for new device From the logs, it appears that the device is being added to the Z-wave network, but that the ISY is unable to create a new primary node. Any ideas? I'm running the following (device and firmware info): I don't have a Polisy or other local polyglot to add an ISY Inventory nodeserver, but would be somewhat surprised if I've hit the 256-node limit. Any other good way to count? In case it might matter, the device that first presented the problem is an alarm.com ADC-SWM150 water meter/shutoff, but I am seeing the same pattern with other devices that I have since tried, including an Aeotech Smart Switch 7. Solutions? ... ideas?

Fair enough. Kind of figured the answer was not until Z-Wave support in Polisy/ISY and then based on a 700 series USB dongle. Agree on speed issue for lighting. Will continue to use Insteon for anything with multi-responder scenes.

Hi Craig, where did you obtain the updated firmware and what is the update mechanism?

I haven't really looked in great detail, but my understanding is that there are some significant advantages both in the key exchange technique employed by S2 (vs. S0) during device enrollment and in the encryption level employed for transport. So ... yes on security against hacking into the over the air communications. My recollection is that there were some pretty widely recognized weaknesses in S0 that basically exposed the network key to any attacker within RF range during device pairing. Out-of-band aspects of the S2 key exchange (device specific codes) are meant to address that. But even with S2 here are published exploits that demonstrate attacks to force a back-compatibility downgrade to S0. No panacea here, obviously, but now that security is more the responsibility of the protocol framework and less an ad hoc vendor-by-vendor thing, it kind of begs the when question for ISY. FWIW, I believe there are also some communication (and therefore power consumption for a battery-powered device) efficiencies due to a single frame transmission [S2] vs. three frames [S0]. To your question of what the ISY would do with a bad frame, I'd like to think that the S2 framework has some basic robustness to denial of service flooding or amplification attacks, but don't really know. For the RF attack surface, there are at least some practical spatial/physical limitations on sources of malicious inbound frames, though the limited channel bandwidth may be easy to saturate even from a small number of in-range transceivers.

I understand that the ISY994 ZW has not, to date, supported the S2 framework for Z-Wave security. Is there a reason? ... will that change? Most every Z-Wave device I have (all series 500 or 700) supports S2, but that support is meaningless without support at the hub as well. I suppose a related question is when will ISY implementations/firmware support Z-Wave Plus V2 (series 700), which requires S2 support?

Thanks. Paired fine once closer to ISY. Strange since every device is ZW+ and NWI is enabled at ISY.

Recently bought a pair of the new Aeotec Smart Switch 7 plug-in type B (ZWA023-A) modules (series 700). After numerous failed pairing/pairing/factory-reset sequences, I finally got one paired. Can't seem to even get the second one fully paired. When the ISY even reacts to the pairing button push (which seems random), it reports errors in the log stream. While there is clearly a communication problem, I don't think it is signal. I've got plenty of repeater pathways in the mesh, and even in the failed partial pairings, end up showing 6 nearby Z-Wave nodes as neighbors. Any thoughts? Several items of note: (1) running 5.3.3 with a series 500 dongle. (2) all Z-Wave nodes (except the two new ZWA023-As) are Z-Wave Plus