Jump to content

Brultech - ISY NodeLink nodes (GEM data) stops displaying data.


mblitz

Recommended Posts

I have an isy994i running v5.0.8 alpha.  I've setup a Raspberry Pi running the latest jessie OS, NodeLink 0.6.7 and emoncms.  I have data from my Brultech GEM feeding NodeLink, and NodeLink pushing data to the isy994i and to emoncms.

 

After a while (around 6 hours), NodeLink is still pushing data to emoncms, but the channel nodes on the isy stopped displaying data.  I don't mean it stops updating. I mean the Current, Power and Total Energy are blank.  Restarting NodeLink doesn't fix the problem, nor does rebooting the isy994i or RPi.  I end up having to delete the NodeLink node server from the isy994i, deleting all the channels, restarting the isy994i, then reinstalling the NodeLink node server.  Then the channels show up again with valid data.

 

This has happened several times.

 

Looking at the isy994i error log, I see the following.  I reinstalled the node server late this morning.  That's probably the 11:12:05 AM messages.  Then around 04:59:47, it appears to have failed.

 

Sat 2017/01/14 11:12:05 AM System -5 Start

Sat 2017/01/14 11:12:09 AM System -110022 /DEF/F6/I1/NLS/EN_US.TXT
Sat 2017/01/14 11:12:15 AM System -170001 [Network] Established
Sat 2017/01/14 04:59:47 PM System -170001 [uDSockets] RSub:32 error:6
Sat 2017/01/14 04:59:52 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:02 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:13 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:23 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:34 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:44 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:00:54 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:05 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:15 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:26 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:36 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:46 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:01:56 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:02:09 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:02:20 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:02:30 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:02:40 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:02:51 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:04 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:14 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:24 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:35 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:45 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:03:55 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:04:06 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:04:16 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:04:26 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:04:41 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:04:51 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:05:02 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:05:12 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:05:23 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:05:40 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:05:50 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:00 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:10 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:21 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:32 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:47 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:06:57 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:07:10 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:07:20 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:07:30 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:07:41 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:07:51 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:03 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:13 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:26 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:36 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:48 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:08:58 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:08 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:19 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:28 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:38 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:49 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:09:59 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:10:10 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:10:20 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:10:30 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:10:41 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:10:52 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:02 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:12 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:23 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:33 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:43 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:11:53 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:04 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:14 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:25 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:35 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:45 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:12:56 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:13:07 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:13:17 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:13:27 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:13:37 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:13:49 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:00 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:10 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:20 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:30 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:42 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:14:52 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:03 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:13 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:23 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:33 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:44 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:15:55 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:05 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:15 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:26 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:36 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:46 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:16:56 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:17:20 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:17:30 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:17:40 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:17:51 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:18:01 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:18:12 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:18:28 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:18:38 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 05:18:49 PM System -170001 UDQ: Queue(s) Full, message ignored
Sat 2017/01/14 04:59:52 PM System -170001 [uDSockets] RSub:32 error:6
Sat 2017/01/14 04:59:57 PM System -170001 [uDSockets] RSub:32 error:6
Link to comment
Share on other sites

Hello mblitz,

 

The errors you see are due to ISY being bombarded with events. The fact that neither rebooting ISY nor NodeLink helps tells me that there's probably a flurry of events coming to ISY in a short amount of time. Is there any way, to check for this hypothesis?

 

With kind regards,

Michel

Link to comment
Share on other sites

Hi Michel,

 

I tested if it was related to the NodeLink.  I pointed NodeLink to an invalid address to throw away isy messages for an hour.  There were no error messages.  As soon as a pointed NodeLink back to my isy994i, I started receiving the "UDQ: Queue(s) Full, message ignored" error log messages.

 

So I think there is a problem with NodeLink.

 

Marty

Link to comment
Share on other sites

Hi Michel,

 

The GEM is in server mode, so I assume NodeLink is polling the GEM.  The frequency is 10s, which is also the update time for emoncms (pushed from NodeLink).

 

The update time on the isy is much slower.  It appears to be updating every 30s.

 

thanks,

marty

Link to comment
Share on other sites

Hi Michel,

 

Yes, i have lots of programs.  Triggered by time of day, motion detectors and alexa (via isy portal).  I also have an Autelis pool control for my Pentair Easy Touch.

 

All are working fine, as far as I know.

 

marty

Link to comment
Share on other sites

Hi Marty,

 

I very much doubt all of them are working simply because when the Queue is full (especially the one in your log), some events will be dropped and not sent to your programs. Also, if the programs are busy, then the queue may get full. So, the best way to test whether or not the issue is the program vs. events, it's best to disable all My Programs by putting a condition that evaluates to false on My Programs folder.

 

With kind regards,

Michel

Link to comment
Share on other sites

Hi Marty,

 

I very much doubt all of them are working simply because when the Queue is full (especially the one in your log), some events will be dropped and not sent to your programs. Also, if the programs are busy, then the queue may get full. So, the best way to test whether or not the issue is the program vs. events, it's best to disable all My Programs by putting a condition that evaluates to false on My Programs folder.

 

With kind regards,

Michel

 

 

Hi Michel,

 

It took a couple of days, but I'm seeing the errors again, and the GEM channel data is blank again.  I haven't disabled my programs yet.  Now that I'm in this state, do you want me to disable the programs and see if I continue to get the errors?

 

marty

Link to comment
Share on other sites

Hi Marty,

 

Thank you. Do you know whether or not NodeLinc polls GEM or is it having GEM publish events using the Realtime mode? In both cases, do you know the frequency?

 

With kind regards,

Michel

 

NodeLink uses the RT mode, so it'll get data at whatever rate the GEM sends.

 

There could potentially be 100+ values that it updates each GEM update (32 channels - A x V x P; 8 counters, 8 temps).  NodeLink only sends changed values to the ISY but it could still be very large.  Nothing I can do there unless the ISY is changed to allow a single bulk data send.

 

When NodeLink encounters a send error, it waits 30s before attempting any new send to the ISY.  So if the ISY is continually getting a full queue from NodeLink, then it's sending the wrong reply on the queue fail (I've seen a number of improper http return values in the 5.XX firmware and mentioned them to Chris).

Link to comment
Share on other sites

 

NodeLink uses the RT mode, so it'll get data at whatever rate the GEM sends.

 

There could potentially be 100+ values that it updates each GEM update (32 channels - A x V x P; 8 counters, 8 temps).  NodeLink only sends changed values to the ISY but it could still be very large.  Nothing I can do there unless the ISY is changed to allow a single bulk data send.

 

When NodeLink encounters a send error, it waits 30s before attempting any new send to the ISY.  So if the ISY is continually getting a full queue from NodeLink, then it's sending the wrong reply on the queue fail (I've seen a number of improper http return values in the 5.XX firmware and mentioned them to Chris).

 

 

I only have 30 channels I'm using.  The GEM is not sending counters or temps.  The GEM actually sends all 32 channels, because apparently NodeLink sends invalid data to emoncms for missing channels.

 

Would it help if I reduce the GEM send rate from 10s to maybe 20?  I've seem the failures occur within a few minutes of setting up the node server on the isy, up to several days before it starts failing.

Link to comment
Share on other sites

That's a question for UDI.  In my opinion you should not have to lower this any - I would hope the ISY could handle this type of load (otherwise we're in trouble with V5 node servers).

 

I have your other various GEM issues fixed (local emoncms address, emoncms node number, I_GEM_C, 0 for null channel values).  No date on a new version but they'll be included.

Link to comment
Share on other sites

That's a question for UDI.  In my opinion you should not have to lower this any - I would hope the ISY could handle this type of load (otherwise we're in trouble with V5 node servers).

 

I have your other various GEM issues fixed (local emoncms address, emoncms node number, I_GEM_C, 0 for null channel values).  No date on a new version but they'll be included.

 

Thanks very much.  I hope I was able to help testing.

Link to comment
Share on other sites

Hi Michel,

 

At this point I see no data for any nodes displayed.  That's the GEM node, lights, sensors, everything.

 

I created a variable, "true" with a value of 1, and was going to use that as the conditional for "My Programs":  if true equal 0

 

But I can't see my variable in the conditional drop-down.

 

I rebooted the isy, and still none of the device statuses showed up.   I deleted the isylink (NodeLink) node server from the isy and rebooted again.  All the device statuses show up now, accept the GEM channel data, which is expected, since I deleted the node server.

 

marty

Link to comment
Share on other sites

Hi Michel,

 

Saturday I reinstalled the isy node server again, and ran for a few hours.  I got lots of the error "System -170001 UDQ: Queue(s) Full, message ignored" messages.  Before it got to the point were I lost status updates, I shutdown NodeLink on my RPi (I actually just redirected to an invalid IP address).  I haven't got any error logged since then, and all my device statuses are updating properly, except GEM channels obviously, which are displaying the last updates.

 

I will redirect NodeLink back to the isy, and wait for a total failure, then send the error logs.

 

marty

Link to comment
Share on other sites

Hi Marty,

 

I think this is just due to the sheer number of events being generated by GEM. Can you reduce the real time interval to 20 seconds?

 

With kind regards,

Michel

 

 

Hi Michel,

 

I already tried that, but still got the errors.  I will try again.

 

marty

Link to comment
Share on other sites

Hi Michel,

 

I've been running GEM with a 20s interval since yesterday, and I'm still getting the errors in the log, but the isy has not locked up.  It is still updating the GEM channel statuses as well as other circuits.

 

So its got that going for it.

 

I will email the error logs.

 

marty

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...