I've recently had two lockups of the same eisy since updating to 14.4-RELEASE-p6. The lockups were roughly 48 hours apart. In each case, the console showed a controller timeout and register dump together with an indication of an unrecoverable I/O failure in the ZFS pool. See below:

After reboot, I ran a zpool status, which reported no errors. It's now been about 36 hours since the last lock up, and I noted the following in eisy-ui:

Is there possibly a memory leak in IoX 6.1.2_7? What do you suggest?

Edited Thursday at 07:11 PM2 days by automatewhatican
fix typos; improve precision

Do a power cycle, that should clear things up

Let me know if the memory usage went down

FYI I'm running 14.3-RELEASE-p5 on a Polisy and also have very high memory usage (96% 3.8 GB / 4.0 GB)

Was wondering the same about a leak.

23 minutes ago, Techman said:

Do a power cycle, that should clear things up

Let me know if the memory usage went down

Done. As expected, down to 24% (1.8 GB / 7.7 GB). I guess we'll have to see if it creeps back up over the next 36 hrs.

The "high memory usage" is a bit of a myth. What's happening is that eisy-ui is only looking at the "Free" (ie: never used) memory. Unix is a memory paging system and uses disk caching (buffers) for speed, so it will use almost all available memory for caching if it's not needed for actual tasks. So memory useage will always appear high. This is my eisy below. Look at the "Mem:" line: the 6828K "Laundry" usage is caching, buffers that ultimately get written out. The memory actually available is usually considered to be "Free" + "Inact"(ive). The screen below that is the "About" screen, reflecting only the 345M as being free.

19 minutes ago, Guy Lavoie said:

The "high memory usage" is a bit of a myth. What's happening is that eisy-ui is only looking at the "Free" (ie: never used) memory. Unix is a memory paging system and uses disk caching (buffers) for speed, so it will use almost all available memory for caching if it's not needed for actual tasks. So memory useage will always appear high. This is my eisy below. Look at the "Mem:" line: the 6828K "Laundry" usage is caching, buffers that ultimately get written out. The memory actually available is usually considered to be "Free" + "Inact"(ive). The screen below that is the "About" screen, reflecting only the 345M as being free.

Thanks, Guy. I understand the up pressure from memory paging and caching. The real problem is the repeated lock ups, requiring hard restart. The eisy-ui's RED symbology for the 95% utilization is what caught my attention. Next time, I'll check the proportion of the pool containing active pages, leaving aside dirty inactive pages.

Try checking it every few hours, to see if there is a progression.

The lockups are unusual. Seems we'd hear other reports if it was an issue with a release.

2 hours ago, Guy Lavoie said:

Try checking it every few hours, to see if there is a progression.

The lockups are unusual. Seems we'd hear other reports if it was an issue with a release.

FreeBSD is bulletproof, and highly stable, that said 14.4-RELEASE will hit its official End of Life (EoL) on December 31, 2026.

After this date, the 14.4 branch will stop receiving critical security advisories or patch updates

It was succeeded by FreeBSD 14.5-RELEASE in September 2026.

So the Polisy will fall farther behind.

24 minutes ago, Guy Lavoie said:

So the Polisy will fall farther behind.

It seems so.

I actually tried to resurrect my Polisy after being in storage for about 8 months. Looks like the mb battery died and it lost all of its settings. All I get is 3 flashing lights.

Guess it's time for the recycle bin.

It should be recoverable, I'd think. It could also be the power supply. I've seen that happen.

6 hours ago, Guy Lavoie said:

Try checking it every few hours, to see if there is a progression.

The lockups are unusual. Seems we'd hear other reports if it was an issue with a release.

Will do, Guy. So far so good; the memory utilization (as indicated in eisy-ui) has been hovering around 25% and about 6GB of free memory is showing in freeBSD system monitor. Agree on unusual. Reliable, non-stop unattended performance is really the isy value proposition, but these recurring lockups are now of concern. I will give it 48 hrs, checking memory; failures appear to occur in early AM. If there is anything else you'd suggest monitoring or checking, let me know.

Edited Friday at 01:45 PM1 day by automatewhatican
clarity

16 hours ago, Guy Lavoie said:

Try checking it every few hours, to see if there is a progression.

The lockups are unusual. Seems we'd hear other reports if it was an issue with a release.

Overnight, memory utilization (as indicated in eisy-ui) spiked to 94%. Less than 0.5 GB remains free per system monitor (189M Active, 700M Inact, 7812K Laundry, 6270M Wired, 465M Free). Nothing strikes me as inherently problematic in that, but it might be notable that someone at UD decided to code the thermometer bar to present in RED rather than BLUE at these levels.

Is there something else I should be checking? ... or just wait for lock up and then pull some log?

There are some overnight tasks that might have used up some of the available memory. 90%+ memory usage is what everyone pretty well sees on a running system. A Polisy or a eisy-lite only has 4 GB of RAM and get along just fine.

As for the lock ups, do you have any particular plugins or other programming that sets your controller apart?

4 minutes ago, Guy Lavoie said:

There are some overnight tasks that might have used up some of the available memory. 90%+ memory usage is what everyone pretty well sees on a running system. A Polisy or a eisy-lite only has 4 GB of RAM and get along just fine.

As for the lock ups, do you have any particular plugins or other programming that sets your controller apart?

On plugins for this eisy, I'm running production versions of OpenWeatherMap 4.0.3 and iAquaLink 3.1.10 plugins and have a production version of Notification 3.6.32 that is UNconnected in PG3. On the ISY side, I have a pretty good set of Z-Wave and Insteon nodes, probably at least 20 of each, although I can do an inventory if helpful. Zwave is mostly AeoTec multisensors and switches and a home energy meter node that has been unpaired for a couple years but for which an inactive node remains. I also have three Honeywell T6 thermostats and a Yale lock. I have about 20 programs; nothing super complicated, but a fair amount of environmental monitoring, and motion or time-cued lighting and thermostat setpoint controls. Insteon is entirely lighting controls. Other than the standard 3am Query and any usual eisy/freeBSD housekeeping, I don't think there is anything particularly noteworthy in the overnight execution.

Z-Wave is an original ZMatter; I've never updated ZMatter firmware. Other than that, packages are all pretty current.

Does anything catch your eye as a good line of inquiry/investigation?

Edited Friday at 03:11 PM1 day by automatewhatican

So nothing out of the ordinary.

The screenshot in your original post seems to indicate a possible hardware issue with the controller's built-in storage controller. mmcsd0 is normally the primary flash storage. It's as if it was not responding and timed out. Hardware failures are rare, but not impossible. If it was a driver issue (from an update) I would think that others would have reported it.

Is your eisy in a confined space? Does it get very hot?

10 minutes ago, Guy Lavoie said:

So nothing out of the ordinary.

The screenshot in your original post seems to indicate a possible hardware issue with the controller's built-in storage controller. mmcsd0 is normally the primary flash storage. It's as if it was not responding and timed out. Hardware failures are rare, but not impossible. If it was a driver issue (from an update) I would think that others would have reported it.

Is your eisy in a confined space? Does it get very hot?

It's in a computer closet with a reasonably controlled environment. Current ambient is only 81.4F. eisy itself feels warm to the touch, not hot.

I don't really have any other suggestions. Hopefully others have suggestions.

11 hours ago, automatewhatican said:

Just for fun, I upgraded to eisy-ui 1.4.2 minutes ago, and my memory usage shows 97%, but the line is blue. I don't know if it's just a change they made to make it feel more normal, or if there's a meaning to the color.