mblitz Posted January 14, 2017 Posted January 14, 2017 I have an isy994i running v5.0.8 alpha. I've setup a Raspberry Pi running the latest jessie OS, NodeLink 0.6.7 and emoncms. I have data from my Brultech GEM feeding NodeLink, and NodeLink pushing data to the isy994i and to emoncms. After a while (around 6 hours), NodeLink is still pushing data to emoncms, but the channel nodes on the isy stopped displaying data. I don't mean it stops updating. I mean the Current, Power and Total Energy are blank. Restarting NodeLink doesn't fix the problem, nor does rebooting the isy994i or RPi. I end up having to delete the NodeLink node server from the isy994i, deleting all the channels, restarting the isy994i, then reinstalling the NodeLink node server. Then the channels show up again with valid data. This has happened several times. Looking at the isy994i error log, I see the following. I reinstalled the node server late this morning. That's probably the 11:12:05 AM messages. Then around 04:59:47, it appears to have failed. Sat 2017/01/14 11:12:05 AM System -5 Start Sat 2017/01/14 11:12:09 AM System -110022 /DEF/F6/I1/NLS/EN_US.TXT Sat 2017/01/14 11:12:15 AM System -170001 [Network] Established Sat 2017/01/14 04:59:47 PM System -170001 [uDSockets] RSub:32 error:6 Sat 2017/01/14 04:59:52 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:02 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:13 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:23 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:34 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:44 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:00:54 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:05 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:15 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:26 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:36 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:46 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:01:56 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:02:09 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:02:20 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:02:30 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:02:40 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:02:51 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:04 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:14 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:24 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:35 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:45 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:03:55 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:04:06 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:04:16 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:04:26 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:04:41 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:04:51 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:05:02 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:05:12 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:05:23 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:05:40 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:05:50 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:00 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:10 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:21 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:32 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:47 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:06:57 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:07:10 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:07:20 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:07:30 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:07:41 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:07:51 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:03 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:13 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:26 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:36 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:48 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:08:58 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:08 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:19 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:28 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:38 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:49 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:09:59 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:10:10 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:10:20 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:10:30 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:10:41 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:10:52 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:02 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:12 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:23 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:33 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:43 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:11:53 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:04 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:14 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:25 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:35 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:45 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:12:56 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:13:07 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:13:17 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:13:27 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:13:37 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:13:49 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:00 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:10 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:20 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:30 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:42 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:14:52 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:03 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:13 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:23 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:33 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:44 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:15:55 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:05 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:15 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:26 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:36 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:46 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:16:56 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:17:20 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:17:30 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:17:40 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:17:51 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:18:01 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:18:12 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:18:28 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:18:38 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 05:18:49 PM System -170001 UDQ: Queue(s) Full, message ignored Sat 2017/01/14 04:59:52 PM System -170001 [uDSockets] RSub:32 error:6 Sat 2017/01/14 04:59:57 PM System -170001 [uDSockets] RSub:32 error:6
Michel Kohanim Posted January 15, 2017 Posted January 15, 2017 Hello mblitz, The errors you see are due to ISY being bombarded with events. The fact that neither rebooting ISY nor NodeLink helps tells me that there's probably a flurry of events coming to ISY in a short amount of time. Is there any way, to check for this hypothesis? With kind regards, Michel
mblitz Posted January 15, 2017 Author Posted January 15, 2017 Hi Michel, I tested if it was related to the NodeLink. I pointed NodeLink to an invalid address to throw away isy messages for an hour. There were no error messages. As soon as a pointed NodeLink back to my isy994i, I started receiving the "UDQ: Queue(s) Full, message ignored" error log messages. So I think there is a problem with NodeLink. Marty
mblitz Posted January 16, 2017 Author Posted January 16, 2017 Also, after the channels stop being updated, when I log into the isy console, I get a pop-up error "Already registered".
Michel Kohanim Posted January 16, 2017 Posted January 16, 2017 Hi Marty, Thank you. Do you know whether or not NodeLinc polls GEM or is it having GEM publish events using the Realtime mode? In both cases, do you know the frequency? With kind regards, Michel
mblitz Posted January 16, 2017 Author Posted January 16, 2017 Hi Michel, The GEM is in server mode, so I assume NodeLink is polling the GEM. The frequency is 10s, which is also the update time for emoncms (pushed from NodeLink). The update time on the isy is much slower. It appears to be updating every 30s. thanks, marty
Michel Kohanim Posted January 16, 2017 Posted January 16, 2017 Hi Marty, Thank you. I am not sure why you would have UDQFull messages with that frequency. Do you also have many other programs that do things based on the values from GEM nodes? With kind regards, Michel
mblitz Posted January 16, 2017 Author Posted January 16, 2017 Hi Michel, I haven't gotten past the point of getting data sent to the isy, so I haven't integrated GEM data into my programs yet. So no, I'm not using GEM data yet. marty
Michel Kohanim Posted January 17, 2017 Posted January 17, 2017 Hi Marty, Thank you. Do you have any other programs? With kind regards, Michel
mblitz Posted January 17, 2017 Author Posted January 17, 2017 Hi Michel, Yes, i have lots of programs. Triggered by time of day, motion detectors and alexa (via isy portal). I also have an Autelis pool control for my Pentair Easy Touch. All are working fine, as far as I know. marty
Michel Kohanim Posted January 17, 2017 Posted January 17, 2017 Hi Marty, I very much doubt all of them are working simply because when the Queue is full (especially the one in your log), some events will be dropped and not sent to your programs. Also, if the programs are busy, then the queue may get full. So, the best way to test whether or not the issue is the program vs. events, it's best to disable all My Programs by putting a condition that evaluates to false on My Programs folder. With kind regards, Michel
mblitz Posted January 19, 2017 Author Posted January 19, 2017 Hi Marty, I very much doubt all of them are working simply because when the Queue is full (especially the one in your log), some events will be dropped and not sent to your programs. Also, if the programs are busy, then the queue may get full. So, the best way to test whether or not the issue is the program vs. events, it's best to disable all My Programs by putting a condition that evaluates to false on My Programs folder. With kind regards, Michel Hi Michel, It took a couple of days, but I'm seeing the errors again, and the GEM channel data is blank again. I haven't disabled my programs yet. Now that I'm in this state, do you want me to disable the programs and see if I continue to get the errors? marty
io_guy Posted January 20, 2017 Posted January 20, 2017 Hi Marty, Thank you. Do you know whether or not NodeLinc polls GEM or is it having GEM publish events using the Realtime mode? In both cases, do you know the frequency? With kind regards, Michel NodeLink uses the RT mode, so it'll get data at whatever rate the GEM sends. There could potentially be 100+ values that it updates each GEM update (32 channels - A x V x P; 8 counters, 8 temps). NodeLink only sends changed values to the ISY but it could still be very large. Nothing I can do there unless the ISY is changed to allow a single bulk data send. When NodeLink encounters a send error, it waits 30s before attempting any new send to the ISY. So if the ISY is continually getting a full queue from NodeLink, then it's sending the wrong reply on the queue fail (I've seen a number of improper http return values in the 5.XX firmware and mentioned them to Chris).
mblitz Posted January 20, 2017 Author Posted January 20, 2017 NodeLink uses the RT mode, so it'll get data at whatever rate the GEM sends. There could potentially be 100+ values that it updates each GEM update (32 channels - A x V x P; 8 counters, 8 temps). NodeLink only sends changed values to the ISY but it could still be very large. Nothing I can do there unless the ISY is changed to allow a single bulk data send. When NodeLink encounters a send error, it waits 30s before attempting any new send to the ISY. So if the ISY is continually getting a full queue from NodeLink, then it's sending the wrong reply on the queue fail (I've seen a number of improper http return values in the 5.XX firmware and mentioned them to Chris). I only have 30 channels I'm using. The GEM is not sending counters or temps. The GEM actually sends all 32 channels, because apparently NodeLink sends invalid data to emoncms for missing channels. Would it help if I reduce the GEM send rate from 10s to maybe 20? I've seem the failures occur within a few minutes of setting up the node server on the isy, up to several days before it starts failing.
io_guy Posted January 20, 2017 Posted January 20, 2017 That's a question for UDI. In my opinion you should not have to lower this any - I would hope the ISY could handle this type of load (otherwise we're in trouble with V5 node servers). I have your other various GEM issues fixed (local emoncms address, emoncms node number, I_GEM_C, 0 for null channel values). No date on a new version but they'll be included.
mblitz Posted January 20, 2017 Author Posted January 20, 2017 That's a question for UDI. In my opinion you should not have to lower this any - I would hope the ISY could handle this type of load (otherwise we're in trouble with V5 node servers). I have your other various GEM issues fixed (local emoncms address, emoncms node number, I_GEM_C, 0 for null channel values). No date on a new version but they'll be included. Thanks very much. I hope I was able to help testing.
Michel Kohanim Posted January 20, 2017 Posted January 20, 2017 Hi io_guy, Thanks for the feedback. Hi Marty, Yes, it's very important to see whether or not programs have any impact. With kind regards, Michel
mblitz Posted January 20, 2017 Author Posted January 20, 2017 Hi Michel, At this point I see no data for any nodes displayed. That's the GEM node, lights, sensors, everything. I created a variable, "true" with a value of 1, and was going to use that as the conditional for "My Programs": if true equal 0 But I can't see my variable in the conditional drop-down. I rebooted the isy, and still none of the device statuses showed up. I deleted the isylink (NodeLink) node server from the isy and rebooted again. All the device statuses show up now, accept the GEM channel data, which is expected, since I deleted the node server. marty
Michel Kohanim Posted January 22, 2017 Posted January 22, 2017 Hi Marty, just to be clear: as long as nodelink is installed in ISY, you don't get any status updates? Can you please send your error log to support@universal-devices.com? With kind regards, Michel
mblitz Posted January 23, 2017 Author Posted January 23, 2017 Hi Michel, Saturday I reinstalled the isy node server again, and ran for a few hours. I got lots of the error "System -170001 UDQ: Queue(s) Full, message ignored" messages. Before it got to the point were I lost status updates, I shutdown NodeLink on my RPi (I actually just redirected to an invalid IP address). I haven't got any error logged since then, and all my device statuses are updating properly, except GEM channels obviously, which are displaying the last updates. I will redirect NodeLink back to the isy, and wait for a total failure, then send the error logs. marty
Michel Kohanim Posted January 23, 2017 Posted January 23, 2017 Hi Marty, I think this is just due to the sheer number of events being generated by GEM. Can you reduce the real time interval to 20 seconds? With kind regards, Michel
mblitz Posted January 23, 2017 Author Posted January 23, 2017 Hi Marty, I think this is just due to the sheer number of events being generated by GEM. Can you reduce the real time interval to 20 seconds? With kind regards, Michel Hi Michel, I already tried that, but still got the errors. I will try again. marty
mblitz Posted January 24, 2017 Author Posted January 24, 2017 Hi Michel, I've been running GEM with a 20s interval since yesterday, and I'm still getting the errors in the log, but the isy has not locked up. It is still updating the GEM channel statuses as well as other circuits. So its got that going for it. I will email the error logs. marty
Michel Kohanim Posted January 25, 2017 Posted January 25, 2017 Hi Marty, Thanks so very much! Would you now increase to 30 seconds? io_guy, Do you know how many status updates are sent to ISY for each channel? With kind regards, Michel
io_guy Posted January 28, 2017 Posted January 28, 2017 io_guy, Do you know how many status updates are sent to ISY for each channel? Previous versions of NodeLink had 3, the latest version has 4. The values are only sent to the ISY when they change.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.