bpwwer

The indexing in WeatherPoly is zero based while the list in crt.py starts at one. So the index is 1 less than the line #. That probably isn't documented anywhere. While it's possible to add more node types and more data values, it does take a bit of effort and because the way Polyglot and the ISY are currently architected, everything tends to be fairly static. What that means is that adding in something like lake level is probably not something that would be used by very many people (I.E. probably just you at this time). If there was a way to do additions like this with just changes to the configuration, I'd do that. But I currently don't have the time to come up with a way to add user defined nodes/values. BTW, it's just about as easy to hack on the files in a Polisy as it is in a RPi.

And to update the status of the modified weewx-crt extension. Matt rejected the pull request to add this ability to his extension and he had some pretty good reasons. Mainly he thought it was a bit outside the scope of his extension and would rather see it as a separate extension. He also noted that as it's currently written, a problem with the HTTP request could block WeeWx for an extended amount of time which is not a good thing. I understand his concerns and agree that the it should be fixed so that it can't block WeeWX. However, I'm not really using WeeWX and don't really want to try and maintain an extension for it. If someone who is willing to maintain an extension wants to step up, I'd be happy help modify my version of the weewx-crt so that it can be a standalone extension. In the meantime, since this is working, I'll keep my fork up there on github for anyone that needs it.

I'm not seeing the duplicates. I do get updates every few seconds but the messages from WeeWX in the /var/log/messages correspond to the log entries in the WeatherPoly log. Currently, WeatherPoly handles bad data by reporting the error and ignoring it. Anything that's not numeric is considered bad data.

I'm not sure I understand what you mean by two sets of identical log entries. Do mean the whole block between "Got some WeeWx data" and "POST /weewx HTTP/1.1 200" is duplicated? That would imply that you are actually getting two requests from WeeWx (or have two copies running maybe). Or are you referring to the two lines above? Those are normal. The first is my code saying it's telling Polyglot to send an update to the ISY and the second is Polyglot saying it is sending the up to the ISY. I tend to sometimes over log things. I've started converting my node servers over to have user configurable logging along with some other improvements but I've not yet cleaned up this node server.

I'm seeing this in the log (/var/log/messages) when it starts Jan 24 10:59:12 piserver weewx[17156]: crt: service version is 0.20 Jan 24 10:59:13 piserver weewx[17156]: crt: realtime txt output goes to /var/tmp/realtime.txt Jan 24 10:59:13 piserver weewx[17156]: crt: realtime txt output posted to http://192.168.92.11:8080/weewx You could add a log output to the post_data function in crt.py so that it outputs something every time it tries to send the data. Maybe something like. def post_data(self, url, data): try: r = requests.post(url = url, data = data) if r.status_code != 200: logerr("crt: Failed to post realtime data to %s: %d" % (url, r.status_code) else: loginf("crt: Successfuly posted realtime data to %s" % url) except Exception, e: logdbg("crt: Exception while sending realtime data to server.") Then I see messages like: Jan 24 11:27:27 piserver weewx[19639]: crt: crt: Sucessfuly posted realtime data to http://192.168.92.11:8080/weewx

Sorry, I should have mentioned that it was on the branch. I tend to use branches when making changes to something to keep those changes separate from the original. I think crt.py supports both methods to specify realtime.txt. Looks like it first tries the option 'filename' and if that doesn't exist, it tries 'realtime_txt'. You should be able to copy the crt.py over the existing one. On my system it is located in /usr/share/weewx/user. In the weewx config file you did add the 'realtime_url = http://<ip>:8080/weewx' right? That's what tells crt.py to post the data. You can also check the weewx log and it should report an error if the post request failed. It will also send info to the debug log for some errors, but I'm not sure how to enable the debug logging. My weewx log is going to /var/messages I believe.

Ok, I have something that appears to be working for me. I forked Matt's repository to my github at https://github.com/bpaauwe/weewx-crt.git and made the changes on a new branch called 'post' There's a couple of changes vs. what I posted above, mainly I had to add one line to initialize the self.realtime_url variable and I added some error trapping so that it doesn't crash if the URL is not valid. I then modified WeatherPoly so that it does the .decode() on the data. That's it. weewx seems to be sending the data and the node server is getting it and parsing it. If you can confirm that the crt changes work, I'll send them over to Matt to see if he wants to include them.

I enjoy working on this type of stuff. I did find my weewx install and added the extension. Looks like my patch isn't quite correct as it crashes weewx. But I can debug it now that it's running.

I didn't send it, I just used git to create a diff file that could be sent. That was the quickest way I know to create something I could post. I can send it to Matt via a git pull request but we should first make sure it actually works. I had weewx installed on a RPi at one point, I should see if I still do. Do you have a clone of Matt's crt git repository? If so, you copy the patch I posted above and apply it to the code with the 'git am <file>' command. This will actually create a new commit in your local clone with this change. The byte encoding isn't really a problem and it's probably not worth the effort to write a parser for the .xml version since we can decode and parse the text file fine. I suppose it might make it a bit easier to configure since the .xml includes labels so the mapping would be from xml label to node instead of field number to node.

It didn't look like it was URL encoded in the log output message. I think it's actually just the data type. the 'b' indicates binary (or byte) data so it's really an array of bytes not a character string. The .decode() function may just be converting it from byte array to string. It may be curl or it just may be how http.server presents the data. It's ok if we have to do that conversion in the node server. Looking at the crt.py code, it looks like it would be really easy to modify it to do a POST of the realtime data. I just did a quick hack and think this might be close From a9503b460602139bf4cb03892cf0ffe2293b155c Mon Sep 17 00:00:00 2001 From: Bob Paauwe <bpaauwe@yahoo.com> Date: Thu, 23 Jan 2020 10:06:23 -0800 Subject: [PATCH] POST realtime.txt to URL Add the ability to send the realtime data to a URL via a POST request. Signed-off-by: Bob Paauwe <bpaauwe@yahoo.com> --- bin/user/crt.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/bin/user/crt.py b/bin/user/crt.py index 51f039d..52e26e9 100644 --- a/bin/user/crt.py +++ b/bin/user/crt.py @@ -32,6 +32,7 @@ Other parameters may be specified to control units and output: [CumulusRealTime] realtime_txt = /path/to/realtime.txt realtime_xml = /path/to/realtime.xml + realtime_url = http://server:port/url sunbird = /path/to/sunbird.txt realtimegauges_txt = /path/to/realtimegaugesT.txt date_separator = / @@ -81,6 +82,7 @@ from a number of live Cumulus sites: import math import time import syslog +import requests from distutils.version import StrictVersion import weewx @@ -464,11 +466,14 @@ class CumulusRealTime(StdService): self.gauges_txt = d.get('realtimegauges_txt') if self.gauges_txt: loginf("gauges output goes to %s" % self.gauges_txt) + if self.realtime_url: + loginf("realtime txt output posted to %s" % self.realtime_url) if (not self.realtime_txt and not self.realtime_xml and not self.sunbird_txt and - not self.gauges_txt): + not self.gauges_txt and + not self.realtime_url): loginf("aborted: no output files specified") return @@ -538,6 +543,9 @@ class CumulusRealTime(StdService): if self.gauges_txt: self.write_data(self.gauges_txt, self.create_gauges_string(data)) + if self.realtime_url: + self.post_data(self.realtime_url, + self.create_realtime_string(data)) except Exception, e: # FIXME: make this catch more specific logdbg("crt: Exception while handling data: %s" % e) weeutil.weeutil.log_traceback('crt: **** ') @@ -548,6 +556,11 @@ class CumulusRealTime(StdService): f.write(data) f.write("\n") + def post_data(self, url, data): + r = requests.post(url = url, data = data) + if r.status_code != 200: + logerr("crt: Failed to post realtime data to %s: %d" % (url, r.status_code)) + # convert from database unit system to specified units def _cvt(self, from_v, to_units, obs, group): if self.db_us is None: -- 2.14.5

I just did a quick look and now I'm not sure the python http server class handles PUT requests. But it's probably still worth experimenting with to see if that's true or not. However, I think that POST requests should work as well either by making sure the content-type is plain text vs url encoded or by checking the content-type in the node server and doing the url decode if it is url encoded. If I recall correctly, the weewx architecture allows for add-ons fairly easy so once you have your python script working you can try to get it added to the weewx code. That would be a nice addition.

That's great! Good job figuring it out. Looking at the options you're using with curl, it does make sense that the data.decode() is needed. The '-d' option is specifying that it sends the data as URL encoded. I was, again, making a bad assumption and assuming the raw data file was being sent. Looking at curl, maybe using the --data-binary option would send it without the URL encoding. However, what might be better would be to get the data via a PUT request instead of POST request. Then you'd use the --upload-file option in curl. That would just send the file as is. I guess the next step is to get the file contents sending automatically somehow. What you end up doing there will really determine what needs to be done in the node server. There was some thought behind having multiple nodes with only a few values per node. The two man reasons are: There is a limit as to how many values you can have on a single node. I think if all possible values were placed in one node, it would exceed that. Also, at a certain point the admin console ability to arrange them to make them all visible seems to fail. There have been discussions on how best to organize nodes so that third party apps may be able to make use of them. In this case, I can designate a node as having values for a specific sensor type (like temperature, humidity, pressure, etc.). When you start mixing various sensor types in one node, how to represent that to a third party app? That's not to say that this is the best solution, I'm open to discussions about this. In fact, for the various weather service node servers, I've tried to define a standard set of values for current conditions and another standard set of values for forecast data. Those nodes are mixed types but maybe by standardizing them it will make it possible to interchange one weather service node server for another. Maybe something similar can be done for weather station node servers.

Ok, so I guess I have to give up on trying to reuse one of the existing parser functions and add one specifically for the weewx data. I just pushed a first attempt at that, see if it works any better.

I'm sorry I haven't really put any thought into this, but a quick look suggested that it would be fairly simple assuming that the data matched what Cumulus was sending. Bad assumption on my part. But that file format looked familiar and looking a little closer, it looks similar to the Meteobridge data. So maybe it's possible to use that parser. The Meteobridge parser requires that you configure a mapping of nodeserver driver to a field number where the field number is an index into a space separated list of data. In fact, looking at the file you posted above and the list I have in the POLYGLOT_CONFIG.md file, they seem to match. So I think calling the meteobridge parser will work so I just pushed that change. You will need to configure the custom parameters with the node to field mapping before it will do anything useful. I.E. temperature-main -> 2 humidity-main -> 3 temperature-dewpoint -> 4 Also, you can modify the code there on the Polisy yourself and try things. You just need to restart the node server after you make changes. There is one issue to be aware of. Polyglot won't update the node server from the store if you've modified any of the files. So to get back to the originals use the 'git checkout <filename>' command. So if you modify the weatherstation.py file you'll need to do git checkout weatherstation.py and it will remove all your changes and allow Polyglot to update from the store again. It is possible to do node server development on the Polisy, but it does take some work to set it up.

No, you're not doing anything wrong, just the code still isn't right. I've not loaded up this node server to test. Basically, just don't have the time right now. I can push a new version but you're correct, it's likely failing the test. That part was just mostly cut-n-paste from the code that processes the GET requests. Probably should be: def process_post_data(self, path, data): if 'weewx' in path: self.cumulus(data) return Do you know how to manually update the node server without waiting for the store to update? You're using a Polisy right? If you ssh into your Polisy you can do the following to update the node server: cd /var/polyglot/nodeservers/WeatherPoly (I think that's the right path) sudo git pull origin master You can then just restart the node server using the dashboard. No need to reboot the Polisy

My mistake. I do this all the time. The function call should be "self.process_post_data" not just "process_post_data". Pushed a new version with the fix. Your RPi has both Python 2.7 and and Python 3.7. To access Python 3.7 use python3. Most of the node servers are written for Python 3.

It can now accept a POST request (I think). I added that in and pushed version 0.1.7. It also redirects a POST request URL with weewx in it to the cumulus parser to parse the data in the body. So if you can post the CRT file to http://<ip>:8080/weewx it might actually work.

Currently, WeatherPoly only handles GET requests and parses the data from the URL query params. But it certainly would be possible to handle POST requests also and parse data from whatever was sent. The easiest type of data to deal with would be JSON as that can be deserialized into a python object and manipulated directly. I believe it can do the same with XML but I'm less familiar with that. Parsing plain text would be a bit more effort but doable.

No, I've not looked into supporting weewx, but if it does have a way of sending it's data to URL, then it's certainly possible to do so. I think what you're missing is that Polyglot/Polisy needs to be running a node server that defines the nodes for the ISY and does the mapping. I've written a node server (WeatherPoly) that acts as a simple web server so it can accept data via a URL from other weather software, creates the nodes on the ISY and then maps the weather data that it gets from the URL to the ISY node values. It does allow you to configure what data from the URL is mapped to the various nodes on the ISY. Right now, if it sees the string "weewx" in a url, it will simply dump what it got to the log file as I don't know what format the data's in so I can't parse it out and map it. If you're able to run this and get the contents of the log file to me, I can add the mapping to make it work. My guess is that you'll need to configure a URL in weewx to send the data to if WeatherPoly is running, you'd specify the IP address of the Polyglot RPi/Polisy with weewx in the url. For example: http://192.168.1.100:<portnum>/weewx/ using the <portnum> you define in the WeatherPoly custom parameters (I believe the default is port 8080).

Already did, thanks for the suggestion!

That's a good question and I agree with you about the quality of the documentation given that I'm responsible for a number of node servers with poor documentation. In my case, and I suspect others as well, because we aren't currently being compensated, it tends to be harder to justify the time for it. Having a reasonably good template for documentation and have it be a gate (I.E. done to some standard) before allowing something to be added to the store would help. I'm sure we'll get there eventually. In the meantime, if you see something you'd like better documented in any of my node servers, let me know. I'm always willing to add things to my todo list

The formula is correct but there was a problem with the wind speed units. I was doing the wrong conversion and using a wind speed value that was almost 4x what it should be. So even a very small amount of wind will cause it to calculate a much lower apparent temperature than it should. I also have the Windchill calculation using the wrong units so it's off my a couple of degrees too. Both will be fixed in the next release.

Where do you see "Apparent Temperature" in the app? I can't find it. I do find a "Feels Like" which is sometimes used interchangeably but sometimes isn't. I'm using a formula for apparent temperature that matches the one here: https://www.vcalc.com/equation/?uuid=081fa2f3-1d2b-11e6-9770-bc764e2038f2 The WeatherFlow feels like temperature is defined here: https://weatherflow.github.io/SmartWeather/api/derived-metric-formulas.html#feels-like-temperature

The node server does create the nodes in order but doesn't have any control over the order they actually get added to the ISY. I've added a task to add the leading zero to my todo list.

Opera, just started happening yesterday for me too.