TJF1960 Posted July 15, 2023 Posted July 15, 2023 Hello @bpwwer, Running eisy IoX 5.6.3 and PG3x 3.1.37. Vue has been an amazing NS. The only issue I have is when eisy is rebooted, not all data populates in IoX. I have to restart the NS then all data will populate. I have, from AC, tried query but that does not populate the missing data. I have even waited a day or two after eisy reboot but some data is still not populated. Everything works fine before the NS manual restart. The missing data is in the fields of Status of each note, and all of the KWH locations in each node. What logs might be beneficial? Is this a known issue, or maybe just my issue? Is there a way to restart either the NS or pg3x remotely, either by USI mobile or by a program in IoX? Thanks! -Tim
bpwwer Posted July 15, 2023 Posted July 15, 2023 Is this something new that happening or something that has always happened? Can you clarify a couple of things for me. First, when you say reboot eisy, what specifically are you doing; rebooting the hardware (Reboot button under eisy System Settings) or restarting the IoX (Restart IoX button)? what data is populating? Most useful would be to set the node server log level to debug and then reboot the eisy. After things have stabilized, the restart the node server manually. Download and send me the log package.
ShawnW Posted July 15, 2023 Posted July 15, 2023 This has often happened for me as well on my Polisy, after Upgrades Packages. It's NOT just that some data is missing, but I also can't control them.
TJF1960 Posted July 16, 2023 Author Posted July 16, 2023 17 hours ago, bpwwer said: Download and send me the log package. Where do I send the log package please?
TJF1960 Posted July 16, 2023 Author Posted July 16, 2023 (edited) @Geddy, do you happen to know the upload size limit when sending a PM? The smallest log file is 1.4Mb and it is hanging up, the other I need to upload is 5.5Mb. @bpwwer I have been unable to attach the logs to a PM to you. Suggestions? Thanks! Tim Edited July 16, 2023 by TJF1960
Geddy Posted July 16, 2023 Posted July 16, 2023 @TJF1960 not sure. But shouldn’t be an issue for those sizes. Make sure you zip them just to be sure they’re compressed as much as possible. Other option would be to open a support ticket (https://www.universal-devices.com/my-tickets) and attach them there, then PM @bpwwer your ticket number. 1
TJF1960 Posted July 16, 2023 Author Posted July 16, 2023 @bpwwer, I was unable to attach any of the zipped logs to your pm so as Geddy suggested, I opened a ticket and was able to upload all logs. Ticket #22024. Thanks! Tim
bpwwer Posted July 16, 2023 Posted July 16, 2023 2 hours ago, TJF1960 said: @bpwwer, I was unable to attach any of the zipped logs to your pm so as Geddy suggested, I opened a ticket and was able to upload all logs. Ticket #22024. Thanks! Tim I reviewed the logs and responded in the ticket. Basically, I don't see anything wrong in the log. 1
TJF1960 Posted July 16, 2023 Author Posted July 16, 2023 Hi @bpwwer, I added updated logs and more info to the ticket. Thanks! Tim
bpwwer Posted July 17, 2023 Posted July 17, 2023 @TJF1960 I don't know if you saw the update in the ticket, but I made a small change to the node server and released it as version 1.0.24. I've been testing with that version and I'm not seeing any issues when restarting the eisy. The Emporia devices that I have do populate values on restart and appear to be updating properly. If you're still seeing the issue with nodes not populating with this version, then more investigation will be required. 1
TJF1960 Posted July 18, 2023 Author Posted July 18, 2023 (edited) Hi @bpwwer, I did see your response last night and updated the Vue NS with the new version, thank you. I was not able to reboot the eisy hardware until just now. I am sorry to say that I am seeing the same thing after hardware reboot. Is it possible that the NS is sending the data too early or too fast in the IoX boot up process? Or maybe there is something on my end that is causing it? Is there anything you can think of I should try? Thank you! Tim Edited July 18, 2023 by TJF1960
bpwwer Posted July 18, 2023 Posted July 18, 2023 I'm not able to reproduce the issue you're seeing. When I reboot my eisy, the node server starts and data is shown and updating. Did you try a query this time? If so, did it help now or not? We have other reports of this same issue but with different node servers. I made a change that I think will help. When you have a chance, try version 1.0.25
TJF1960 Posted July 19, 2023 Author Posted July 19, 2023 Hi @bpwwer, query did not work on 1.0.24, I will download 1.0.25 tonight but wont be able to test until after work tomorrow. Thank you for your perseverance Bob! -Tim
TJF1960 Posted July 21, 2023 Author Posted July 21, 2023 Hi @bpwwer, I installed 1.0.25, after reboot of eisy I am sorry to say I did not see a change. Query did not help either. What is the recommending method of query at reboot, checking the reboot at start up box or unchecking the box and having a program run a system query 5 or so minutes after reboot? I am reaching for straws here. It must be something on my end here, a program or setting or something. I looked at the error log but didn't see anything out of the ordinary. I have attached it though. Not sure what to try next. A_E.v5.6.2__Sat 2023.07.08 04.52.29 AM.isy
bpwwer Posted July 21, 2023 Posted July 21, 2023 Just checking the query at reboot box should be enough. So that I'm clear. After the reboot, you have device nodes with blank fields. (you had screen shots attached to the ticket, right?). If you either right click -> query or press the query button on the node, it does not populate those blank fields, correct? Then if you restart the node server, all those blank fields are populated without you having to do anything else. If the above is accurate, then I don't think the problem is the node server. As I mentioned, we have other reports of the same behavior but for other node servers. The node server would have to not send data for the fields to remain blank, and based on the node server logs, that's not true. The node server is sending the data but IoX is either not accepting or not seeing it. The fact that it doesn't seem to accept data even later (when you force a query), is what's really strange. I think we're going to have to figure out how to reproduce on one of our (UDI developer's) systems to be able to debug it. I'm not giving up yet.
bpwwer Posted July 22, 2023 Posted July 22, 2023 Hi @TJF1960 If you get a chance to do test, I'd like to try the following. 1. reboot the eisy 2. after everything starts and the AC is showing blank fields for the Vue nodes try the following from a browser window http://<eisy ip>:8080/rest/nodes/n013_187658 You may be able to use "eisy.local' instead of the <eisy.ip> It should prompt you for your Iox username and password. Copy the results to post here. Then restart the node server and do the same thing. Thanks.
TJF1960 Posted July 22, 2023 Author Posted July 22, 2023 (edited) @bpwwer, Thanks Bob, interesting results below. Here is the capture after hardware reboot using "eisy.local": <nodeInfo> <node flag="128" nodeDefId="outlet"> <address>n013_187658</address> <name>WR Bench Charger</name> <family instance="13">10</family> <parent type="3">2249</parent> <type>0.0.0.0</type> <enabled>true</enabled> <deviceClass>0</deviceClass> <wattage>0</wattage> <dcPeriod>0</dcPeriod> <startDelay>0</startDelay> <endDelay>0</endDelay> <pnode>n013_187658</pnode> </node> <properties> <property id="CPW" value="653" formatted="0.0653 kW" uom="30" prec="4" name=""/> <property id="GV1" value="117" formatted="0.0117 kWh" uom="33" prec="4" name=""/> </properties> </nodeInfo> This is after Vue NS restarting: <nodeInfo> <node flag="128" nodeDefId="outlet"> <address>n013_187658</address> <name>WR Bench Charger</name> <family instance="13">10</family> <parent type="3">2249</parent> <type>0.0.0.0</type> <enabled>true</enabled> <deviceClass>0</deviceClass> <wattage>0</wattage> <dcPeriod>0</dcPeriod> <startDelay>0</startDelay> <endDelay>0</endDelay> <pnode>n013_187658</pnode> </node> <properties> <property id="CPW" value="656" formatted="0.0656 kW" uom="30" prec="4" name=""/> <property id="GV1" value="273" formatted="0.0273 kWh" uom="33" prec="4" name=""/> <property id="GV2" value="668" formatted="0.0668 kWh" uom="33" prec="4" name=""/> <property id="GV3" value="2261" formatted="0.2261 kWh" uom="33" prec="4" name=""/> <property id="ST" value="1" formatted="On" uom="25" name=""/> </properties> </nodeInfo> Edited July 22, 2023 by TJF1960
bpwwer Posted July 22, 2023 Posted July 22, 2023 When you got the results with just the 2 properties showing, was the AC showing those same values in the two fields? Or were all the fields still blank?
TJF1960 Posted July 22, 2023 Author Posted July 22, 2023 @bpwwer, only those two fields in AC were populated, the rest were blank.
bpwwer Posted July 24, 2023 Posted July 24, 2023 We've been discussing this and the latest thoughts are that IoX may be clear out fields while PG3 is trying to set them and if the timing is just right what you're seeing could happen. If/When you get a chance, can you turn off the query at restart in IoX (uncheck the checkbox on the AC configuration page) and reboot. And again, after everything is started, check http://eisy.local:8080/rest/nodes/n013_187658. We're looking to see if properties are still missing and/or if there are blank fields for that node. 1
TJF1960 Posted July 25, 2023 Author Posted July 25, 2023 (edited) Hi @bpwwer, I only have time to run once right now, all fields for all vue node are populated after hardware reboot with query turned off. I want to run it a couple of times to be sure but this is the first time all fields have been populated after hardware reboot. Before hitting restart on vue node server: <nodeInfo> <node flag="128" nodeDefId="outlet"> <address>n013_187658</address> <name>WR Bench Charger</name> <family instance="13">10</family> <parent type="3">2249</parent> <type>0.0.0.0</type> <enabled>true</enabled> <deviceClass>0</deviceClass> <wattage>0</wattage> <dcPeriod>0</dcPeriod> <startDelay>0</startDelay> <endDelay>0</endDelay> <pnode>n013_187658</pnode> </node> <properties> <property id="CPW" value="0" formatted="0 kW" uom="30" name=""/> <property id="GV1" value="0" formatted="0 kWh" uom="33" name=""/> <property id="GV2" value="0" formatted="0 kWh" uom="33" name=""/> <property id="GV3" value="2528" formatted="0.2528 kWh" uom="33" prec="4" name=""/> <property id="ST" value="0" formatted="Off" uom="25" name=""/> </properties> </nodeInfo> after hitting restart on NS: <nodeInfo> <node flag="128" nodeDefId="outlet"> <address>n013_187658</address> <name>WR Bench Charger</name> <family instance="13">10</family> <parent type="3">2249</parent> <type>0.0.0.0</type> <enabled>true</enabled> <deviceClass>0</deviceClass> <wattage>0</wattage> <dcPeriod>0</dcPeriod> <startDelay>0</startDelay> <endDelay>0</endDelay> <pnode>n013_187658</pnode> </node> <properties> <property id="CPW" value="0" formatted="0 kW" uom="30" name=""/> <property id="GV1" value="0" formatted="0 kWh" uom="33" name=""/> <property id="GV2" value="0" formatted="0 kWh" uom="33" name=""/> <property id="GV3" value="2528" formatted="0.2528 kWh" uom="33" prec="4" name=""/> <property id="ST" value="0" formatted="Off" uom="25" name=""/> </properties> </nodeInfo> Edit: I was able to run this test one more time last night with the same results. With query box in AC unchecked all node fields populate as they should. When the query box is checked some fields are missing. You guys are on to something for sure. Thanks! Tim Edited July 25, 2023 by TJF1960
bpwwer Posted July 25, 2023 Posted July 25, 2023 When the query all runs at startup, one of things it does is clear the status values and then queries the node server to re-populate them. So the theory was that something like this might be happening: node server starts and starts adding/checking nodes. - gets the node values from IoX - It then starts comparing and updating as necessary. While this is happening, the query is called for this node and IoX clears all the status values, including the ones that were just updated. At startup, a lot is going on so the queries are slow/delayed so they align with the node servers starting time wise. Now that we have an idea about why this is happening, we should be able to come up with a way to fix it. 1
TJF1960 Posted July 26, 2023 Author Posted July 26, 2023 Hi @bpwwer, makes sense. Question, Nightly query and or pressing the query button in the node page in AC must act differently, it must not clear the values like the query at startup does? Also, I wonder why pressing the query or even when the nightly query runs those blank fields never populate? I have gone days without restarting the NS after a hardware reboot and the blank fields stay blank until the NS is rebooted. Thank you for all your help in tracking this down. -TIm
bpwwer Posted July 26, 2023 Posted July 26, 2023 15 hours ago, TJF1960 said: Hi @bpwwer, makes sense. Question, Nightly query and or pressing the query button in the node page in AC must act differently, it must not clear the values like the query at startup does? Also, I wonder why pressing the query or even when the nightly query runs those blank fields never populate? I have gone days without restarting the NS after a hardware reboot and the blank fields stay blank until the NS is rebooted. Thank you for all your help in tracking this down. -TIm The query at start is happening while all the node servers are starting. When PG3 starts all the node servers at once, it generates a lot of traffic and load on the IoX. So the query is likely taking more time. In your case, all the variables align so that the query and node server start for Vue intersect. A nightly query doesn't have to deal with node servers starting so you won't encounter the issue. Why a later query (or even later value updates from the node server) don't update the fields later is a very good question and I don't yet have an answer. Possibly the query marks a value when it clears it and is then waiting for the query to the node server to return a value. If the node server doesn't respond to that specific query, the value is locked waiting for something that never happens. After reporting these latest results, I haven't heard anything from the IoX developer yet.
Recommended Posts