Displaying individual variables "as devices" (sort of like virtual devices)

[markv58]

@Teken Try this, go to the Virtual folder and delete all .db files in the folder then restart the nodeserver. This shouldn't affect any of the settings in the nodes. Restart the nodeserver.

[tmorse305]

 

7 hours ago, markv58 said:

@tmorse305 That can happen on very rare occasions, I believe that is fixed solid in the next update which will show up shortly. The .db files need to be deleted to clear the error. You can do that manually or use the button in each node to do that after the update.

v1.0.14 is pushed.

v1.0.14 solved  the issue.  Its been running correctly for a few hours.  Thank you.

[markv58]

Be absolutely certain that the variable you are pulling from exists! If it does not exist there will most likely be an error that will crash the nodeserver. I will add a routine to check that the variable is valid when I get back to my desk tomorrow evening and run some tests to see what error I need to look for. 

[Teken]

@markv58

I went for broke and deleted the entire node server from the Polisy. Either there's still garbage left in the Polyglot or there are some bugs in the 1.0.14 release. Let me start by asking some basic questions because it seems every release either removes a requirement or adds something that is unknown.

- Precision Value: Do I need to set the ISY to any precision value as all of mine are set to 1.

- Left Buttons: In several releases we had to press the left hand button on each selection for the system to save the option. Do I need to press on the left hand side buttons as I see the *System Busy* now as I make a selection.

- Maximum Nodes: I asked this question several posts back and never received an answer. Does the virtual node server have a limit on how many nodes can be created? As I have 42 nodes all getting set up now and feel this might be one of the issues here. 

- Raw / Precision: As of this writing this function doesn't seem to work even when I started off with a single node.

[markv58]

8 hours ago, Teken said:

Precision Value: Do I need to set the ISY to any precision value as all of mine are set to 1.

Prec 0 or 1, it can handle both.

 

8 hours ago, Teken said:

Left Buttons: In several releases we had to press the left hand button on each selection for the system to save the option. Do I need to press on the left hand side buttons as I see the *System Busy* now as I make a selection.

Those button have given me problems on all nodeservers. Setting the value from the drop down should be enough to set the value and it should change in the panel above. Hitting the button should only resend the set value command.Your confirmation is the proper setting in the reporting panel, that's why I added them. You may or may not see the Busy flag pop on.

 

8 hours ago, Teken said:

Maximum Nodes: I asked this question several posts back and never received an answer. Does the virtual node server have a limit on how many nodes can be created? As I have 42 nodes all getting set up now and feel this might be one of the issues here.

I have no limits applied and have gotten no answer as to if there is a limit of nodes or node types allowed and I have asked. I suppose you could have so many nodes pushing and pulling that it could bog down the polyglot with all of that activity but it seems unlikely on Polisy as that thing has some power.

 

8 hours ago, Teken said:

Raw / Precision: As of this writing this function doesn't seem to work even when I started off with a single node.

I test everything on 2 RPi polyglot and 1 Polisy and everything is working properly. Raw to Prec seems to be needed on any variable that is set the Prec 1.

 

Other things:

 

Make sure the variable exists that you are pulling from, I will add a safety check to ensure that is doesn't try to update from a non-existent variable.

 

You can only pull from 1 variable in each node, I will add a safety check to disallow 2 pull actions.

 

If you send me a log dump I will look through it and see if things are working properly, no sensitive data is in the logs. Redact anything you think necessary and DM or email markv58git@gmail.com.

 

Teken, thanks for all your input and driving the Temperature nodes to this point! The Energy node may possibly give you just as much grief but hopefully not. I plan to get the whole thing ready and tested before release so maybe the bugs will be less and there will be only a couple of profile updates.

 

[Teken]

3 hours ago, markv58 said:

Prec 0 or 1, it can handle both.

 

Those button have given me problems on all nodeservers. Setting the value from the drop down should be enough to set the value and it should change in the panel above. Hitting the button should only resend the set value command.Your confirmation is the proper setting in the reporting panel, that's why I added them. You may or may not see the Busy flag pop on.

 

I have no limits applied and have gotten no answer as to if there is a limit of nodes or node types allowed and I have asked. I suppose you could have so many nodes pushing and pulling that it could bog down the polyglot with all of that activity but it seems unlikely on Polisy as that thing has some power.

 

I test everything on 2 RPi polyglot and 1 Polisy and everything is working properly. Raw to Prec seems to be needed on any variable that is set the Prec 1.

 

Other things:

 

Make sure the variable exists that you are pulling from, I will add a safety check to ensure that is doesn't try to update from a non-existent variable.

 

You can only pull from 1 variable in each node, I will add a safety check to disallow 2 pull actions.

 

If you send me a log dump I will look through it and see if things are working properly, no sensitive data is in the logs. Redact anything you think necessary and DM or email markv58git@gmail.com.

 

Teken, thanks for all your input and driving the Temperature nodes to this point! The Energy node may possibly give you just as much grief but hopefully not. I plan to get the whole thing ready and tested before release so maybe the bugs will be less and there will be only a couple of profile updates.

 

@markv58

Inbox is full . ..  ?

Regardless, here is a snippet of what the Virtual Node Server logs indicate. As of this writing it has stop processing any further inbound data. I spent a few hours last night just restarting the server just to see what the results were. At some point it would just stall and declare the error you see in this image capture. Some other thoughts and insight below:

Process Queue: I think one of the problems for me at least in my environment at different times many 1 wire sensors will change state. My suggestion is to incorporate some kind of staged timing where a person can define how fast data is seen, ingested, and processed. I've worked with several companies to incorporate what I could best describe as a staggered offset. Essentially, if say my system has 32 sensors instead of blasting the controller with one giant REST call. The vendor has incorporated a method to send the channels in chunks. ex. 1-6 send, wait X seconds, 7-12 send, wait X seconds, etc until all 32 channels have been sent. I think a more reasonable approach for any vendor is to process based on a state of change.

Unfortunately for what ever reason many of these companies don't do this?!? 

Process Termination: I'm not sure how you're handling the connection to and from the controller. But, is it possible some of this has to be with running out of socket connections? Perhaps worded in another way if there is a *Best Practice* of connecting and gracefully disconnecting is such a method being used now? I know from past experience depending upon how the connection is made whether it be polling, pushing, or subscribing. Each method can have an impact on the controller when the environment is at a large scale like mine. I honestly don't believe many people run out of I/O's or processes for the controller in normal situations. But when you're a power user and have hundreds of programs, network resources, and inbound network traffic via REST. All of this has a negative impact on the controller to be able to complete other system wide tasks.

It's been quite the honor to play a small role in the development process so - Thank You!

[markv58]

@Teken I can see that the problem is one of the nodes is pulling from a variable does not exist. That will get fixed.  I can't dive in any deeper for a couple of hours but it’s nothing serious. 

[Teken]

6 minutes ago, markv58 said:

@Teken I can see that the problem is one of the nodes is pulling from a variable does not exist. That will get fixed.  I can't dive in any deeper for a couple of hours but it’s nothing serious. 

No worries, I got another ball to play with . . . ? ? ?

[Teken]

12 minutes ago, markv58 said:

@Teken I can see that the problem is one of the nodes is pulling from a variable does not exist. That will get fixed.  I can't dive in any deeper for a couple of hours but it’s nothing serious. 

@markv58

To be clear when you say a variable doesn't exist what does that mean exactly? As I created state variables which span a range of 1~229. Does the current system need to see an actual numeric value in the controllers state variable table? As only 40 something have real values in place from different hardware appliances. The rest are empty as I haven't enough time to set everything up to send and populate those state variables. 

[gviliunas]

I think that I am seeing the same thing as @Teken

State variable 155 is certainly defined in my ISY and also is to only one I'm using in this NS

2020-07-23 12:07:16,349 Controller polyinterface DEBUG Virtual:pullFromID: Pulling from http://192.168.1.70/rest/vars/get/2/155/

2020-07-23 12:07:16,382 Controller polyinterface DEBUG Virtual:pullFromID: b'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="155"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 12:05:30</ts></var>'

2020-07-23 12:07:16,388 Controller polyinterface INFO Virtual:pullFromID: ['b\'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="155"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 12:05:30</ts></var>\'']

2020-07-23 12:07:16,401 Controller polyinterface ERROR polyinterface:write: Exception in thread Controller:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner

self.run()

File "/usr/local/lib/python3.7/threading.py", line 870, in run

self._target(*self._args, **self._kwargs)

File "/var/polyglot/.local/lib/python3.7/site-packages/polyinterface/polyinterface.py", line 854, in _parseInput

self.longPoll()

File "./Virtual.py", line 57, in longPoll

self.nodes[node].getDataFromID()

File "./Virtual.py", line 805, in getDataFromID

self.pullFromID(_type, self.action1id)

File "./Virtual.py", line 822, in pullFromID

LOGGER.info('Init = %s Prec = %s Value = %s',_value[1], _value[2], _value[3])

IndexError: list index out of range

[Teken]

26 minutes ago, gviliunas said:

I think that I am seeing the same thing as @Teken

State variable 155 is certainly defined in my ISY and also is to only one I'm using in this NS

2020-07-23 12:07:16,349 Controller polyinterface DEBUG Virtual:pullFromID: Pulling from http://192.168.1.70/rest/vars/get/2/155/

2020-07-23 12:07:16,382 Controller polyinterface DEBUG Virtual:pullFromID: b'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="155"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 12:05:30</ts></var>'

2020-07-23 12:07:16,388 Controller polyinterface INFO Virtual:pullFromID: ['b\'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="155"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 12:05:30</ts></var>\'']

2020-07-23 12:07:16,401 Controller polyinterface ERROR polyinterface:write: Exception in thread Controller:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner

self.run()

File "/usr/local/lib/python3.7/threading.py", line 870, in run

self._target(*self._args, **self._kwargs)

File "/var/polyglot/.local/lib/python3.7/site-packages/polyinterface/polyinterface.py", line 854, in _parseInput

self.longPoll()

File "./Virtual.py", line 57, in longPoll

self.nodes[node].getDataFromID()

File "./Virtual.py", line 805, in getDataFromID

self.pullFromID(_type, self.action1id)

File "./Virtual.py", line 822, in pullFromID

LOGGER.info('Init = %s Prec = %s Value = %s',_value[1], _value[2], _value[3])

IndexError: list index out of range

This must be a Canadian thing as its apparent @markv58 doesn't know how to deal with the beavers  eh . .  . ?

[markv58]

I don't know why that is happening but I do know how to stop it. Got held up on an IT install, on the road home. finally.  I'll try to do the fix en route and update.

[markv58]

@Teken @gviliunas I removed the offending line and triggered an update. I have no way to test so update and let me know. No need for a profile update just a nodeserver restart.

That may move the error further down the code or it may have been the LOGGER printing before the return values were set.

[gviliunas]

No Joy with version 1.0.15

 

2020-07-23 16:19:50,881 Controller polyinterface DEBUG Virtual:pullFromID: b'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="154"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 11:55:36</ts></var>'

2020-07-23 16:19:50,887 Controller polyinterface INFO Virtual:pullFromID: ['b\'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="154"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 11:55:36</ts></var>\'']

2020-07-23 16:19:50,888 Controller polyinterface DEBUG Virtual:pullFromID: /2/

2020-07-23 16:19:50,903 Controller polyinterface ERROR polyinterface:write: Exception in thread Controller:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner

self.run()

File "/usr/local/lib/python3.7/threading.py", line 870, in run

self._target(*self._args, **self._kwargs)

File "/var/polyglot/.local/lib/python3.7/site-packages/polyinterface/polyinterface.py", line 854, in _parseInput

self.longPoll()

File "./Virtual.py", line 57, in longPoll

self.nodes[node].getDataFromID()

File "./Virtual.py", line 805, in getDataFromID

self.pullFromID(_type, self.action1id)

File "./Virtual.py", line 825, in pullFromID

if command1 == '/2/' : _newTemp = int(_value[3])

IndexError: list index out of range

-----------------------------------------------------------------------------------------------------------------------------------------------

 

Just to be sure I have this configured correctly...

 

My temperature sensor is updating State Variable 154 with a Fahrenheit temperature.

This NS is configured with a temperaturec node associated with State Variable 155

The NS configuration is to pull from SV 154 and push to SV 155 after converting from Raw and Converting to degrees C

 

Is this correct?  

Virtual_logs_2020-7-23_162507.zip

[markv58]

@gviliunas Good log info! The error got kicked down the road, now I can resolve this. The data did pull correctly so it must be a timing thing. Back at my desk in 3 hours. Should not take long, way too bouncy to try a fix on the laptop. 

[gviliunas]

@markv58 You are too dedicated. Take your time.  ??

[markv58]

@Teken @gviliunas How many pulls are you guys doing. I am not getting any errors so it must be an overloading. I will insert a tiny wait like Teken mentioned and see if that clears the bug. 

[gviliunas]

I have only 1 variable configured in this NS in my Polisy. I have 10 other NS and short and long polling for each NS is at default.

[Teken]

@Teken @gviliunas How many pulls are you guys doing. I am not getting any errors so it must be an overloading. I will insert a tiny wait like Teken mentioned and see if that clears the bug. 

Approximately 30-36?


Sent from my iPhone using Tapatalk

[markv58]

@Teken @gviliunas I tried everything I could think of to crash one of my polyglot and no luck, even non-existent variable IDs passed. So all I can figure is there some latency in the system. My ISY is plugged into the router so that may be why.

I added and error handler that should capture errors and not let the nodeserver crash. I can't test it cause I can't get an error so you guys are the testers.

I added another Custom Configuration Parameter: key - parseDelay where you can add a pause, in seconds, to let the data have time to populate. I would start at .1 and work up. I set mine at 1 just to see what happens.  Let me know how it works for you and log bits of errors will be helpful.

Thanks for helping out! I really would like to put the Temperature Nodes to bed so I can start on the Energy Node.

v1.0.16 has been triggered, no update profile, just update and restart.

 

[gviliunas]

@markv58 I tried v1.0.16. This version seems to work and updates the State Variable correctly and the NS Does NOT crash but I am still seeing errors in the log.

First, I tried without a parseDelay then I increased the delay in several steps to finally 4 seconds but still see this error pattern:

020-07-23 21:06:48,320 Controller polyinterface INFO Virtual:pullFromID: ['b\'<?xml version="1.0" encoding="UTF-8"?><var type="2" id="154"><init>0</init><prec>1</prec><val>-16</val><ts>20200723 11:55:36</ts></var>\'']

2020-07-23 21:06:52,398 Controller polyinterface DEBUG Virtual:pullFromID: Parse delay: 4

2020-07-23 21:06:52,399 Controller polyinterface ERROR Virtual:pullFromID: An error occured during the content parse: list index out of range

 

 

Virtual_logs_2020-7-23_210925.zip

[tmorse305]

I am still running v1.0.15, I have 12 nodes ( all value=temperature).  It has been up and running for 12 hours.  Maybe the crash is related to temperaturec?  I'm not using that.

[markv58]

@tmorse305 I'm running all nodes with no errors, mind boggled.

@Teken @gviliunas v1.0.17 adds some logger info to help sort out what is going on. Update, run for a few minutes and give me some log info. Also can you tell me what Polyglot version you are on, Thanks.

If I understand correctly, the values are updating properly but there is still an error reported, is this correct?

[gviliunas]

@markv58   Ah Ha!  With the parseDelay =0, I updated to v1.0.17

The error pattern that I was seeing previously is not occurring now. BUT, before Virtual NS made its first pull from State Variable 154, I set the variable to a large number to verify the calculations were working. I think the problem may be with the "Highest" temperature calculation or with the temperaturec conversion. The other values are changing, as expected, but Highest is always = 0 degrees C

All other displayed values seem correct and are updating.

Virtual_logs_2020-7-24_081905.zip

[markv58]

@gviliunas Great! I still don’t know what was going on. This morning looking thru your log it seemed that you were on a different version, some of the info was not there that should have been. Maybe there was a glitch in the update. 
 

I tested pulling from a non existing variable and the data came back as all 0 with no errors so that may be the lowest temp problem there. When you reset the statistics, the lowest is set to 129 and highest to -60 or -30 depending on the c or f node so that any temp has an extreme to compare to at first run after the reset.