f11

Ah - you've set your system to update on 'loop' (~3 sec period) while mine is set to update on 'archive' (300 sec period). Ok, so the WP log message regarding wind direction isn't really an 'error', its just flagging what it considers bad data. In that case, I've run across zero issues so far. weewx-crt starts and stops gracefully in tandem with weewx (which I would expect), and other than logging activity just hums along in the background. So far, things are working extremely well.

Yes, 'the whole block between "Got some WeeWx data" and "POST /weewx HTTP/1.1 200" is duplicated'. The two blocks are identical in all regards, except for their timestamps, which only differ in msec. Checked for multiple instances of weewx using 'ps aux' and 'htop' -> nope, just one that's visible ... checking for multiple crt extensions in weewx using wee_extensions --list -> nope, just one is listed. Did an update and dist-upgrade of the weewx RPi just in case - it goes long periods without restarting or being updated) - then rebooted Polisy, and rebooted ISY so everything is starting out in a somewhat known state. Everything works like a damn - except for the two blocks of identical log entries. Weird. Is there a way to temporarily disable weewx-crt without stopping-starting everything? If I have some kind of double-instance action going on, maybe I can shut down the one I can see to help me find the one I can't. Since you're not seeing this double-log effect, its obviously at my end. I'll keep looking. But otherwise, everything looks great with WeatherPoly and your mod'd version of weewx-crt : updates come in with only slight latency from the update of realtime.txt to arrival in WP's log, and the data transfers across to ISY smoothly and appears on the ISY screen as expected. The only issue I've seen so far is the failed conversion of realtime.txt 'none/NULL/- - -/N NE E SW etc' strings to values in WP, clearly an artifact of dealing with the incoming file fields instead of live data. Although I've only seen it show up in wind direction (both ordinal and bearing) custom parameter fields, its likely to be common to all converted fields. I was going to test that next, just assigning random fields to various custom parameters to see if WP can handle unexpected inputs gracefully.

Yup, consistently getting two sets of identical log file entries for each update, varying only by time stamp in the msec field. Also, wind direction, both ordinal and bearing, are having problems converting ordinals (SE, WSW, - - - , etc) and bearings=NULL / - - - into floats, understandably. 2020-01-24 14:20:21,995 [Thread-1 ] [INFO ] - Set wind driver GV0 to NULL 2020-01-24 14:20:21,995 [Thread-1 ] [DEBUG] - setDriver failed 7 -> wind could not convert string to float: 'NULL'

Downloaded the .zip file from post, copied your crt.py over Matt's, restarting weewx and waiting ... and messages, updates, and all the rest !! Here's the WP log (hmm, I got these lines twice ... anyway) : 2020-01-24 13:15:22,190 [Thread-1 ] [DEBUG] Got some WeeWX data 2020-01-24 13:15:22,192 [Thread-1 ] [INFO ] - Set temperature driver ST to 4.3 2020-01-24 13:15:22,193 [Thread-1 ] [INFO ] Updating Driver temperature - ST: 4.3, uom: 4 2020-01-24 13:15:22,195 [Thread-1 ] [INFO ] - Set temperature driver GV0 to -0.9 2020-01-24 13:15:22,196 [Thread-1 ] [INFO ] Updating Driver temperature - GV0: -0.9, uom: 4 2020-01-24 13:15:22,198 [Thread-1 ] [INFO ] - Set humidity driver ST to 69 2020-01-24 13:15:22,200 [Thread-1 ] [INFO ] Updating Driver humidity - ST: 69.0, uom: 51 2020-01-24 13:15:22,202 [Thread-1 ] [INFO ] - Set temperature driver GV1 to 4.3 2020-01-24 13:15:22,202 [Thread-1 ] [INFO ] Updating Driver temperature - GV1: 4.3, uom: 4 2020-01-24 13:15:22,204 [Thread-1 ] [INFO ] - Set light driver GV0 to 109 2020-01-24 13:15:22,205 [Thread-1 ] [INFO ] Updating Driver light - GV0: 109.0, uom: 74 2020-01-24 13:15:22,207 [Thread-1 ] [INFO ] - Set temperature driver GV4 to 20.8 2020-01-24 13:15:22,207 [Thread-1 ] [INFO ] Updating Driver temperature - GV4: 20.8, uom: 4 2020-01-24 13:15:22,209 [Thread-1 ] [INFO ] - Set humidity driver GV0 to 34 2020-01-24 13:15:22,210 [Thread-1 ] [INFO ] Updating Driver humidity - GV0: 34.0, uom: 51 2020-01-24 13:15:22,212 [Thread-1 ] [INFO ] - Set pressure driver GV1 to -2.0 2020-01-24 13:15:22,213 [Thread-1 ] [INFO ] Updating Driver pressure - GV1: -2.0, uom: 25 2020-01-24 13:15:22,215 [Thread-1 ] [INFO ] - Set pressure driver ST to 1003.2 2020-01-24 13:15:22,216 [Thread-1 ] [INFO ] Updating Driver pressure - ST: 1003.2, uom: 117 2020-01-24 13:15:22,218 [Thread-1 ] [INFO ] - Set wind driver GV4 to 0.4 2020-01-24 13:15:22,220 [Thread-1 ] [INFO ] Updating Driver wind - GV4: 0.4, uom: 32 2020-01-24 13:15:22,222 [Thread-1 ] [INFO ] - Set wind driver GV1 to 0.4 2020-01-24 13:15:22,224 [Thread-1 ] [INFO ] Updating Driver wind - GV1: 0.4, uom: 32 2020-01-24 13:15:22,226 [Thread-1 ] [INFO ] - Set wind driver GV0 to SE 2020-01-24 13:15:22,227 [Thread-1 ] [DEBUG] - setDriver failed 11 -> wind could not convert string to float: 'SE' 2020-01-24 13:15:22,229 [Thread-1 ] [INFO ] "POST /weewx HTTP/1.1" 200 - I may have the failed 'wind' issue because I grabbed the wrong value - I see I grabbed the ordinal wind dir, not the bearing.

Yup, I understand the follies of running as root - but when I installed weewx way back when, that's how it got installed and I've been reluctant to make the necesary changes. I usually install into my user folders. Yeah, Polisy requires nodeservers to run as polyglot which is what I've got going. I installed an m-sata 250GB SSD on my weewx RPi over a 2 years ago, worrying about the wear and tear on the micro-SD card with all the file writing and re-writing. So everything but the /boot files are on the SSD. pi@RPi3-3-Weewx:~/crt $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 239314556 7323308 219764996 4% / devtmpfs 467512 0 467512 0% /dev tmpfs 472120 0 472120 0% /dev/shm tmpfs 472120 47700 424420 11% /run tmpfs 5120 8 5112 1% /run/lock tmpfs 472120 0 472120 0% /sys/fs/cgroup /dev/mmcblk0p6 67434 24154 43280 36% /boot tmpfs 94424 0 94424 0% /run/user/1000

My weewx is running as root, so I'm used to using 'sudo' a LOT ... and sometimes forget that some programs are supposed to run as a regular user. I assume that weewx-crt should run as root as well? Just checking for permissions issues. -> Edit: ok, think I know what happened - at 4AM when I last downloaded from your repo, I grabbed the master files, NOT the 'post' branch, although I knew I needed the post files ... why I do these things at that time of day I have NO idea - yup, a git-idiot. After that I did indeed install everything but they're Matt's crt, not yours. I guess that might explain the missing log messages ??. I'll fix this up and report back. -> Ah-ha - I've been using the green 'Clone or Download' button and copying the url line from the drop down - which just downloads the master files. Trying to figure out how to download the branch files.

Got it figured. Didn't know I could use wee_extension installer with a directory of the unpacked weewx-crt files - amazing (!) what you can learn when you read the python code itself Huh ... no updates. Obviously I've forgotten SOMETHING. I just don't know what. [ edit: our messages crossed in posting - just checking my weewx.conf file now ... [Engine] [[Services]] . . . . . . archive_services = weewx.engine.StdArchive, user.crt.CumulusRealTime . . . [CumulusRealTime] filename = /var/tmp/realtime.txt realtime_txt = /var/tmp/realtime.txt # realtime_xml = /var/tmp/realtime.xml realtime_url = http://192.168.1.23:8080/weewx # update file on archive period binding = archive # specify my preferred date stamp field separator date_separator = - # units preferences for realtime.txt values unit_system = METRIC wind_units = meter_per_second temperature_units = degree_C pressure_units = mbar cloudbase_units = meter rain_units = mm Maybe I'm not seeing something obvious? I used ''sudo service weewx stop', waiting a minute, then 'sudo service weewx start'. The status option looked normal. Both the Polisy Log and the WP Log show zero activity from WP after starting. Checking /var/log/messages for weewx and crt statements ... hmm, after weewx and crt are up and running, the log only shows weewx general activity, but crt is never heard from again. I'm going to add print statements into crt.py to see what's going on ... this is TOO strange. Hold on ... just scanning through the 'crt.py' file in /usr/share/weewx/user ... it doesn;t have your changes to class CumulusRealTime(StdService) that I was expecting (I can't see references to the self.realtime_url and other things from your earlier patch post). Could have SWORN I dropped your new crt.py here. So, this must be where I dropped the ball. Give me a few minutes to see if I can unwind this snafu.

WeatherPoly is doing its thing, proved by issuing curl POST request from command line - updates arrive at Polisy as expected and are decoded properly. One thing: if a parameter is '---' in the realtime.txt file, it causes problems for the conversion to float (for wind driver. at least): 2020-01-24 04:25:08,251 [Thread-1 ] [INFO ] - Set wind driver GV0 to --- 2020-01-24 04:25:08,252 [Thread-1 ] [DEBUG] - setDriver failed 11 -> wind could not convert string to float: '---' The data that led to that error was wind-winddir : field 11 (right after the pressure reading 1007.2): pi@RPi3-3-Weewx:/etc/weewx/skins/Seasons $ cat /var/tmp/realtime.txt 24-01-20 04:25:00 -6.2 87 -8.0 0.0 0.0 NULL 0.0 0.0 1007.2 --- 0 m/s C mb mm 0.0 -0.5 0.2 0.2 0.0 20.0 35 -6.2 -2.2 -3.3 00:55 -6.2 04:25 0.0 00:05 1.3 00:35 1007.8 01:40 1007.1 00:25 3.9.2 0 0.0 -6.2 -6.2 NULL 0.0 0 NULL 0.0 0 0 0 --- 926 m -9.1 0.0 NULL 0 I'm probably not getting auto-updates due to fumbling the installation of the modified crt.py file. I'll have to check it in the morning.

As far as I can tell, I made all the changes, restarted weewx and restarted Polisy, then stopped and started WeatherPoly. Logs showing no updates, just this: 2020-01-24 02:05:30,469 [Thread-1 ] [INFO ] Started web server on port 8080 2020-01-24 02:14:46,468 [Thread-1 ] [INFO ] "GET / HTTP/1.1" 200 - 2020-01-24 02:14:46,490 [Thread-1 ] [INFO ] "GET /favicon.ico HTTP/1.1" 200 - 2020-01-24 02:16:11,946 [Thread-1 ] [INFO ] "GET /favicon.ico HTTP/1.1" 200 - Notice nothing came in at 2:10 or 2:15 when the realtime.txt file was updated in /var/tmp. Have confirmed realtime.txt is being updated as usual, but no indications its being sent to Polisy. I must have buggered up the modifications. Remind me - how do I install your update via ssh? Serious brain fog right now ... [ edit just in case I'd messed up the WeatherPoly folder (I was getting punchy and notice 'root' showing up as owner, probably from me using 'sudo' on the command line), I totally uninstalled WP from Polisy and re-installed it from the NodeServer Store. I checked weatherstation.py to make sure it had your most recent changes (like the .decode() call). and then restarted everything. No updates coming in to Polisy. ]

Sorry, I'm a git-idiot - it took me a while (well, quite a while) to find your modifed 'post' version of weewx-crt - I didn't realize it was under Branches. So I cloned your weewx-crt onto my RPi, and not sure what I do now : I assume I just cp your crt.py over the one I installed as part of the extension 3 days ago (yep, its been backed up), and add the necessary config lines into weewx.conf. BTW, the crt.py preamble comments at the top shows both the original 'filename = /path/to/realtime.txt' and a newer 'realtime_txt = /path/to/realtime.txt under two different references to [CumulusRealTime] in weewx.conf. I already had the original reference, but added the new one just in case. I suspect both are probably not needed.

You crashed weewx? Wow, that's actually pretty tough to do - congrats !! Now THAT's a bug! Confirm crt changes work - will do !! I'll get your new code up and running tonight, and thrash at it. I wish I had more experience with python so I could provide more substantive help, but that'll come with time, I guess. I've already learned a dozen new things about the language in the past few days/nights working with WP and weew-crt. Still SO MANY things I have no clue about ...

Yup, I have the crt git repo on the same RPi as weewx, so I can patch in your changes there. (GOTTA learn git, clearly its a lot more useful than I realized!). I can poke and prod the changed crt for issues and I'll get back to you. Gotta say, knowing how many other things you're trying to get done, you've spent WAY more time on this than I ever expected. Thanks, Bob!

So it looks like you submitted a change to Matt's code via git to send the file to an URL, right? (like I said I'm a noob in the git world) Your (mostly) additions from above make sense to me. Has this been formally accepted into the codebase, or is there a testing/vetting process that takes place first? Spent last night learning about http.server, and how to get python to make POST/PUT requests. Quite educational, and it helped me understand your weatherstation.py code a lot better. I was getting the same issues with byte-encoding of the file regardless of what I tried, so I think I better admit I don't understand the underlying process behind sending a text file to a webserver, whether via curl or within python using the requests library. I've been labouring under the impression that a text file could be sent to a webserver as a string, but from my reading this doesn't appear to be possible, or its discouraged (one warning used the words 'strongly encouraged to send as bytes' regarding sending text). I was about to start researching sending the xml file instead, but your changes above may make that unnecessary. [ I love reading clean code like above - white space, easy to read, makes it easy to follow and understand! ]

I added some code to your weatherstation.py file to handle curl PUT requests. From what I read about http.server class, PUT requests are valid and handled like other requests, using do_PUT(). In that function I copied do_POST(), modifying it for put_data and self.process_put_data(), then copying your process_post_data() to process_put_data() as a first step. In process_put_data(), I reverted to your original code: if weewx, self.weewx(data) without the .decode() part. sent file = still the exact same as sent via POST command-line = < curl http://192.168.1.23:8080/weewx --upload-file /var/tmp/realtime.txt > From what I read, in this form curl auto-magically uses a PUT request, and is supposed to send the file as-is. The inserted code: # handle put requests def do_PUT(self): message = "<head></head><body>Successful data submission</body>\n" content_length = int(self.headers['content-Length']) put_data = self.rfile.read(content_length) self.process_put_data(self.path, put_data) self.send_response(200) # OK self.send_header("Content-type", "text/html") self.end_headers() self.wfile.write(message.encode('utf_8')) return # process weewx put data (realtime.txt cumulus-format file of values) def process_put_data(self, path, data): LOGGER.debug('process_put_data(): path=%s' % path) LOGGER.debug('process_put_data(): data=%s' % data) if 'weewx' in path: self.weewx(data) return Polisy's log screen showed that, sure enough that curl command was interpreted as a PUT request (first two DEBUG lines) and was handled by the two new do_PUT() and process_put_data() functions: 2020-01-23 00:37:57,337 [Thread-1 ] [DEBUG] process_put_data(): path=/weewx 2020-01-23 00:37:57,338 [Thread-1 ] [DEBUG] process_put_data(): data=b'23-01-20 00:35:00 -8.4 90 -9.7 0.0 0.0 NULL 0.0 0.0 1014.4 --- 0 m/s C mb mm 0.0 1.3 0.2 0.2 0.0 20.6 36 -8.4 -0.8 -8.3 00:15 -8.6 00:25 0.0 00:05 0.4 00:05 1014.6 00:15 1014.5 00:05 3.9.2 0 0.4 -8.4 -8.4 NULL 0.0 0 NULL 0.0 0 0 0 --- 868 m -11.4 0.0 NULL 0\n' 2020-01-23 00:37:57,339 [Thread-1 ] [DEBUG] Got some WeeWX data 2020-01-23 00:37:57,341 [Thread-1 ] [ERROR] ---------------------------------------- 2020-01-23 00:37:57,342 [Thread-1 ] [ERROR] Exception happened during processing of request from 2020-01-23 00:37:57,343 [Thread-1 ] [ERROR] ('192.168.1.183', 59756) 2020-01-23 00:37:57,356 [Thread-1 ] [ERROR] Traceback (most recent call last): 2020-01-23 00:37:57,358 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 316, in _handle_request_noblock self.process_request(request, client_address) 2020-01-23 00:37:57,359 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 347, in process_request self.finish_request(request, client_address) 2020-01-23 00:37:57,360 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request self.RequestHandlerClass(request, client_address, self) 2020-01-23 00:37:57,361 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__ self.handle() 2020-01-23 00:37:57,362 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 426, in handle self.handle_one_request() 2020-01-23 00:37:57,362 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request method() 2020-01-23 00:37:57,363 [Thread-1 ] [ERROR] File "./weatherstation.py", line 767, in do_PUT self.process_put_data(self.path, put_data) 2020-01-23 00:37:57,364 [Thread-1 ] [ERROR] File "./weatherstation.py", line 811, in process_put_data self.weewx(data) 2020-01-23 00:37:57,365 [Thread-1 ] [ERROR] File "./weatherstation.py", line 846, in weewx fields = data.split(' ') 2020-01-23 00:37:57,366 [Thread-1 ] [ERROR] TypeError: a bytes-like object is required, not 'str' 2020-01-23 00:37:57,367 [Thread-1 ] [ERROR] ---------------------------------------- Unfortunately, the realtime.txt file is STILL being received as URL-encoded, judging by the 'data=b' in my DEBUG line, and the fact that we're back to seeing the 'bytes-like object' objection in the ERROR line at the bottom. That suggests the regular POST request should work if the original file is, as you say, sent as content plain-text. It occurs to me that when this thing is finally working, the request will be made from a python script (i suspect), so that may change how the file is sent since curl will be out of the picture. I'll have to try that next.

Yup, weewx provides a pretty straight forward "extension" capability, which is what the weewx CRT (Cumulus Real Time) extension uses. Once attached, an extension functions like part of the weewx class hierarchy. I'm not sure it'd be too cool for me to pretend to be involved with this effort beyond working with you to debug a prototype interface between WP and weewx via the CRT extension and some baling wire. This is why I figured it'd be great if the CRT author (Matthew Wall, I believe) could either offer an option in CRT to send its realtime.txt file to an IP:Port as part of the write/update process, or at least a function that issues an alert whenever realtime.txt has been updated so that 3rd party scripts or executables could act on it. Obviously, whatever might be offered needs to be as lightweight as possible to avoid negatively impacting weewx performance. But I agree - having some way to auto-send a standardized file like realtime.txt to a webserver (not necesssarily the same one weewx sends its FTP web files to) would make a lot of folks pretty happy, not just weewx users. What's even MORE interesting is that this afternoon I found what appears to be a very recent update (like from 20Dec19) to CRT which offers to create not only the cumulus formatted realtime.txt file, but also the same data (I think) in realtime.xml and a couple other optional formats. I haven't installed it yet so not sure if there are gotcha's in those features. <https://github.com/matthewwall/weewx-crt/blob/master/bin/user/crt.py> Been a busy day of snow shovelling here, so I'm just getting started on looking into curl PUT requests.

I'll start investigating using PUT instead of POST, and adjust my version of WP in Polisy accordingly. I'd like the data transfer to be as "clean" as it can be, direct from the weewx-crt extension file to WP node-server if possible, hopefully reducing opportunities for bugs to creep in. If PUT can clean up the transfer, then that would be my preference, too. For auto-sending to Polisy, I'm looking at a short python script using FileSystemEventHandler as the constructor for a custom handler. The example I'm looking at makes sense and I'm familiar with the concepts. Basically my handler will wait to be notified of a modification to realtime.txt (which is re-written every 5 min as part of the VantagePro2 archive process), the just send it to Polisy/weewx. If this can be done with PUT, then you'll have a known entity to work with in WP (a simple text file of space separated values). I would be even better if the crt extension author would add an option to send the file to a webserver (a configurable IP and port, and maybe a choice of sending protocols) as one of the extension features. I haven't attempted to contact the author yet, so that might be a long shot. Having the values all on one page isn't crucial, since the purpose of having them available to ISY is to use them in home automation, not building a pretty weather dashboard (especially in ISY's old-school-like UI). Still, it seems an unnecessary load on Polisy and ISY to have all these nodes piling up.Then again, I guess its just small memory allocations and intra-device messaging, so perhaps the loading is a lot less than I think it is.

You're not gonna believe it, but I got it working! But what a dog's breakfast. I inserted a log of LOGGER.debug statements in your wp.py code, so I could see what was happening to the data passed in from the curl POST request from the command line. First, the realtime.txt data file as written to my RPi3 "disk", the the curl POST request that sent it: pi@RPi3-3-Weewx:~ $ cat /var/tmp/realtime.txt 22-01-20 01:00:00 -1.6 79 -4.8 0.4 0.4 135 0.0 0.0 1002.2 SE 0 m/s C mb mm 1.6 0.0 0.2 0.2 0.0 20.1 36 -1.6 -1.4 -0.8 00:05 -1.6 00:55 0.9 00:15 3.1 00:15 1002.1 00:35 1002.0 00:25 3.9.2 0 1.3 -1.6 -1.6 NULL 0.0 0 135 0.0 0 0 0 SE 1095 m -4.5 0.0 NULL 0 pi@RPi3-3-Weewx:~ $ curl -d @/var/tmp/realtime.txt http://192.168.1.23:8080/weewx <head></head><body>Successful data submission</body> Here's the process_post_data() function with LOGGER's and slightly modified. The data sent by curl arrive at the Polisy server as an encoded string, so I decoded it before sending it to weewx(): def process_post_data(self, path, data): # LOGGER.debug('process_post_data(): path=%s' % path) # LOGGER.debug('process_post_data(): data=%s' % data) if 'weewx' in path: data_b = data.decode() self.weewx(data_b) return I admit, this is sort of a hack as I don't really understand what I'm doing here: I thought what I sent WAS a string ... but whatever for now. I didn't modify your weewx() code except to insert LOGGER statements to see what was going on: def weewx(self, data): LOGGER.debug('Got some WeeWX data') # LOGGER.debug('weewx(): datatype = %s' % type(data)) # LOGGER.debug('weewx(): data = %s' % data) fields = data.split(' ') LOGGER.debug('weewx(): fields = %s' % fields) LOGGER.debug('weewx(): self.node_map = %s' % self.node_map) for f in self.node_map: LOGGER.debug('weewx(): f = %s' % f) try: i = int(f) except: continue m = self.node_map[f] LOGGER.debug('weewx(): m = %s self.node_map[f] = %s' % (m, self.node_map[f])) try: LOGGER.info(' - Set %s driver %s to %s' % (self.node_map[f]['node'], self.node_map[f]['driver'], fields[i])) self.nodes[m['node']].setDriver(m['driver'], float(fields[i])) except Exception as e: LOGGER.debug(' - setDriver failed %s -> %s %s' % (f, m['node'], str(e))) return After restarting WP, nothing happened except the errors no longer displayed in the WP log screen. Then I remembered your comment earlier today about nothing really happening until I entered some customer parameter key:value pairs in the Configuration screen. For weewx-crt data, I needed to match your key names to the data field 0-based number. So following the advice under 'acuparse', I matched up four values from the front end of the data: temperature-main:2, temperature-dewpoint:4, humidity-main:3, and pressure-sealevel:10. I restarted WP, gave everything a minute to settle, then re-sent the curl POST from above. The WP log screen showed: 2020-01-22 00:53:39,273 [Thread-1 ] [DEBUG] Got some WeeWX data 2020-01-22 00:53:39,274 [Thread-1 ] [DEBUG] weewx(): fields = ['22-01-20', '00:50:00', '-1.5', '79', '-4.7', '0.0', '0.0', 'NULL', '0.0', '0.0', '1002.1', '---', '0', 'm/s', 'C', 'mb', 'mm', '1.5', '-0.2', '0.2', '0.2', '0.0', '20.1', '36', '-1.5', '-1.2', '-0.8', '00:05', '-1.4', '00:45', '0.9', '00:15', '3.1', '00:15', '1002.1', '00:35', '1002.0', '00:25', '3.9.2', '0', '0.9', '-1.5', '-1.5', 'NULL', '0.0', '0', 'NULL', '0.0', '0', '0', '0', '---', '1095', 'm', '-4.1', '0.0', 'NULL', '0'] 2020-01-22 00:53:39,275 [Thread-1 ] [DEBUG] weewx(): self.node_map = {'10': {'node': 'pressure', 'driver': 'GV0', 'units': 'I_MB'}, '4': {'node': 'temperature', 'driver': 'GV0', 'units': 'I_TEMP_C'}, '3': {'node': 'humidity', 'driver': 'ST', 'units': 'I_HUMIDITY'}, '2': {'node': 'temperature', 'driver': 'ST', 'units': 'I_TEMP_C'}} 2020-01-22 00:53:39,276 [Thread-1 ] [DEBUG] weewx(): f = 10 2020-01-22 00:53:39,276 [Thread-1 ] [DEBUG] weewx(): m = {'node': 'pressure', 'driver': 'GV0', 'units': 'I_MB'} self.node_map[f] = {'node': 'pressure', 'driver': 'GV0', 'units': 'I_MB'} 2020-01-22 00:53:39,277 [Thread-1 ] [INFO ] - Set pressure driver GV0 to 1002.1 2020-01-22 00:53:39,278 [Thread-1 ] [INFO ] Updating Driver pressure - GV0: 1002.1, uom: 117 2020-01-22 00:53:39,281 [Thread-1 ] [DEBUG] weewx(): f = 4 2020-01-22 00:53:39,282 [Thread-1 ] [DEBUG] weewx(): m = {'node': 'temperature', 'driver': 'GV0', 'units': 'I_TEMP_C'} self.node_map[f] = {'node': 'temperature', 'driver': 'GV0', 'units': 'I_TEMP_C'} 2020-01-22 00:53:39,282 [Thread-1 ] [INFO ] - Set temperature driver GV0 to -4.7 2020-01-22 00:53:39,283 [Thread-1 ] [INFO ] Updating Driver temperature - GV0: -4.7, uom: 4 2020-01-22 00:53:39,286 [Thread-1 ] [DEBUG] weewx(): f = 3 2020-01-22 00:53:39,287 [Thread-1 ] [DEBUG] weewx(): m = {'node': 'humidity', 'driver': 'ST', 'units': 'I_HUMIDITY'} self.node_map[f] = {'node': 'humidity', 'driver': 'ST', 'units': 'I_HUMIDITY'} 2020-01-22 00:53:39,288 [Thread-1 ] [INFO ] - Set humidity driver ST to 79 2020-01-22 00:53:39,290 [Thread-1 ] [INFO ] Updating Driver humidity - ST: 79.0, uom: 51 2020-01-22 00:53:39,291 [Thread-1 ] [DEBUG] weewx(): f = 2 2020-01-22 00:53:39,292 [Thread-1 ] [DEBUG] weewx(): m = {'node': 'temperature', 'driver': 'ST', 'units': 'I_TEMP_C'} self.node_map[f] = {'node': 'temperature', 'driver': 'ST', 'units': 'I_TEMP_C'} 2020-01-22 00:53:39,293 [Thread-1 ] [INFO ] - Set temperature driver ST to -1.5 2020-01-22 00:53:39,294 [Thread-1 ] [INFO ] Updating Driver temperature - ST: -1.5, uom: 4 2020-01-22 00:53:39,296 [Thread-1 ] [INFO ] "POST /weewx HTTP/1.1" 200 - When I went to my ISY WeatherPoly Summary, the data was there. Command line curl updates subsequently sent to Polisy were almost instantly visible on the WP Nodes screen and in ISY. So at least we know this setup can work. My data.decode() solution is a hack as I said, so I'm sure you can do this a lot more elegantly than me, now that we know what's going on. I have to wonder if the way I do the curl POST request is responsible for this decoding, and if changing the curl statement would make the decoding unnecessary - I don't know enough on that. Now the only thing I need to figure out is how to get weewx to either automatically send that realtime.txt file to Polisy server using something similar to curl, perhaps within the crt extension if I can reach the author for help; or how to have a python script be aware of when the file is changed and then resend it to the Polisy server - perhaps a separate thread that just monitors the timestamp and resends the file when the timestamp changes? Is there anyway to have all of the values WP can forward to ISY appear on one ISY summary screen, instead of on separate screens (well, ok - the -main and -dewpoint temperatures are already on one screen)? I suspect the answer is no, as each measurement falls into a group with the same basic units of measurement.

Will do. I thought you had it with meteobridge. what does the data that comes in for meteobridge look like anyway? I was thinking the process_post_data() function could massage the weewx crt data format before being passed to the existing meteobridge() function, but I don't know (yet) what that might be. Still no-go. WP log displays (I just grabbed the last few lines): 2020-01-21 22:16:56,926 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request method() 2020-01-21 22:16:56,927 [Thread-1 ] [ERROR] File "./weatherstation.py", line 751, in do_POST self.process_post_data(self.path, post_data) 2020-01-21 22:16:56,927 [Thread-1 ] [ERROR] File "./weatherstation.py", line 784, in process_post_data self.weewx(data) 2020-01-21 22:16:56,928 [Thread-1 ] [ERROR] File "./weatherstation.py", line 817, in weewx fields = data.split(' ') 2020-01-21 22:16:56,929 [Thread-1 ] [ERROR] TypeError: a bytes-like object is required, not 'str' 2020-01-21 22:16:56,930 [Thread-1 ] [ERROR] ---------------------------------------- Using the python console, this approach worked. On looking up the error, it seems we're bumping into a bytes-vs-strings issue that crops up between python v2 and v3.

Nope, no cigar. I thought I'd send the curl request before making any changes to the WP custom parms, sort of as a benchmark. Sending the same curl request as previously resulted in curl: (52) Empty reply from server at the terminal, and in 2020-01-21 16:58:02,163 [NodeServer] [INFO ] WeatherPoly Node Server Started. 2020-01-21 16:58:02,167 [Thread-1 ] [INFO ] Started web server on port 8080 2020-01-21 16:58:14,640 [Thread-1 ] [ERROR] ---------------------------------------- 2020-01-21 16:58:14,641 [Thread-1 ] [ERROR] Exception happened during processing of request from 2020-01-21 16:58:14,642 [Thread-1 ] [ERROR] ('192.168.1.183', 40172) 2020-01-21 16:58:14,657 [Thread-1 ] [ERROR] Traceback (most recent call last): 2020-01-21 16:58:14,658 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 316, in _handle_request_noblock self.process_request(request, client_address) 2020-01-21 16:58:14,659 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 347, in process_request self.finish_request(request, client_address) 2020-01-21 16:58:14,660 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request self.RequestHandlerClass(request, client_address, self) 2020-01-21 16:58:14,661 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__ self.handle() 2020-01-21 16:58:14,661 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 426, in handle self.handle_one_request() 2020-01-21 16:58:14,662 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request method() 2020-01-21 16:58:14,663 [Thread-1 ] [ERROR] File "./weatherstation.py", line 751, in do_POST self.process_post_data(self.path, post_data) 2020-01-21 16:58:14,664 [Thread-1 ] [ERROR] File "./weatherstation.py", line 784, in process_post_data self.meteobridge(data) 2020-01-21 16:58:14,664 [Thread-1 ] [ERROR] File "./weatherstation.py", line 792, in meteobridge fields = data[key][0].split(' ') 2020-01-21 16:58:14,665 [Thread-1 ] [ERROR] TypeError: 'int' object is not subscriptable 2020-01-21 16:58:14,666 [Thread-1 ] [ERROR] ---------------------------------------- at the Polisy WP log screen. So reading WP process_post_data() function code in Polisy, realtime.txt file is sent to the self.meteobridge(data) function. Inside the function, it breaks apart the key:value pair as: 'key' is the incoming file, and 'value' is the file content (have i got that right?). Then you try to break the string of values into 'fields', so that the values in the data stream can be paired with fields like temperature or humidity and so on, depending on order in the stream. The error is just saying it can't iterate over 'key' which is an integer. When I followed the data through the Polisy WP code, and inserted debug LOGGER lines in the function to see what it gets as 'data' and what it extracts as 'key', the LOGGER reported: 2020-01-21 17:50:00,026 [Thread-1 ] [DEBUG] meteob - data=b'21-01-20 17:45:00 -0.9 80 -4.0 0.0 0.0 NULL 0.0 0.0 1003.6 --- 0 m/s C mb mm 0.1 -1.3 0.2 0.2 0.0 19.6 35 -0.9 -2.2 1.5 15:40 -8.7 07:15 0.4 11:50 1.3 11:50 1007.7 04:50 1003.7 17:30 3.9.2 0 0.0 -0.9 -0.9 NULL 0.0 0 NULL 0.0 0 0 0 --- 1076 m -3.4 8.3 NULL 0' 2020-01-21 17:50:01,052 [Thread-1 ] [DEBUG] meteob - key=50 So 'key' is coming in as the first char of 'data', which is '2' (ascii 50) of the date field, generating the error. The way I'm POSTing the realtime.txt seems to be missing something that can act as the key into the ' ' separated fields. [ edit1: not sure what the 'b' means in " data=b'21-01-20 ...' ". The LOGGER statement I inserted was 'LOGGER.debug('meteob - data=%s' % data)'

I'll do the update right away. Thx for mentioning how to re-establish the update process - I was already messing around in weatherstation.py (adding some logger statements to see what your server is receiving from my submissions) after saving the original under a diffferent name, so Polisy has probably already recognized an attempt at modification. Just did the git checkout and then the git pull, and I'm back in sync with your work. Still have to do the node config mapping as you suggested. You rock, Bob !

Update worked as expected, restarted WP node-server. Resent curl string. What I'm sending in the file isn't what Cumulus function is expecting: you're apparently looking for key value pairs, and all I'm sending is the values, space separated. When I checked the log, I got a around 200 of the same INFO statements, if key in self.node_map: ... etc etc ... else: LOGGER.info('map has %d entries, but not %s' % (len(self.node_map), key)) taking the form, 'map has 0 entries, but not [various numbers like 50, 49, 45, 48 ... ]' I'll experiment with a modified realtime.txt file just to see what happens, although I suspect the problem is the missing keys. Thanks again!

I can appreciate you're busy - I honestly didn't expect so many replies from you in such a short time - so thanks for all you've done so far!! Didn't know about that update procedure using 'git pull', so I'll give it a try (I'm a git-noob.) I can wait on further development if you're under the gun on other projects. You do what ya gotta do - we're good!

+1 for python 3 Updated WeatherPoly, rebooted Polisy just in case. Resent at the command line: curl -d @/var/tmp/realtime.txt http://192.168.1.23:8080/weewx Response on the terminal was: <head></head><body>Successful data submission</body> Polisy's WeatherPoly log reported just this one line: 2020-01-21 10:46:44,424 [Thread-1 ] [INFO ] "POST /weewx HTTP/1.1" 200 - and nothing more. Looking at your weatherstation.py code: def process_post_data(self, path, data): c = path.split('?') if len(c) > 1: #data = urllib.parse.parse_qs(c[1]) if 'weewx' in c[0]: self.cumulus(data) return you're ensuring 'weewx' was in the submitted URL string before sending the data (the realtime.txt data file) on to self.cumulus(data). But the first thing the cumulus function does is log 'Got some cumulus data'. I never see that, so I assume the test 'if len(c) > 1:' failed. If there was no '?' character in the URL string, len(c) returns 1 which indeed fails the test, and the data never reaches the cumulus function. Do I need to send my POST request in a different format?

Sorry, meant to show you the file content. Here's a listing of the file: pi@RPi3-3-Weewx:/etc/weewx $ cat /var/tmp/realtime.txt 21-01-20 00:30:00 -5.2 85 -7.3 0.0 0.0 NULL 0.0 0.0 1006.8 --- 0 m/s C mb mm 0.0 0.2 0.2 0.2 0.0 20.6 34 -5.2 -1.6 -4.2 00:05 -4.9 00:25 0.0 00:05 0.0 00:05 1007.0 00:05 1006.8 00:15 3.9.2 0 0.0 -5.2 -5.2 NULL 0.0 0 NULL 0.0 0 0 0 --- 966 m -8.0 0.0 NULL 0

Ok - installed crt v0.18 extension to weewx. made sure it was installed where I expected files to show up (I run weewx as root) with correct ownership and permissions. Restarted weewx and all looked good, so at least crt didn't bork my weather data. The data file is created and updated in RPi folder /var/tmp/realtime.txt. I sent a curl POST request from the command line to Polisy as: curl -d @/var/tmp/realtime.txt http://192.168.1.23:8080/weewx The response at the command line was: curl: (52) Empty reply from server where I believe '(52)' is the number of values present in the realtime.txt file. Watching the WeatherPoly log, the following was eventually displayed: 2020-01-21 00:41:32,225 [Thread-1 ] [ERROR] ---------------------------------------- 2020-01-21 00:41:32,226 [Thread-1 ] [ERROR] Exception happened during processing of request from 2020-01-21 00:41:32,227 [Thread-1 ] [ERROR] ('192.168.1.183', 58448) 2020-01-21 00:41:32,242 [Thread-1 ] [ERROR] Traceback (most recent call last): 2020-01-21 00:41:32,243 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 316, in _handle_request_noblock self.process_request(request, client_address) 2020-01-21 00:41:32,244 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 347, in process_request self.finish_request(request, client_address) 2020-01-21 00:41:32,245 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request self.RequestHandlerClass(request, client_address, self) 2020-01-21 00:41:32,246 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__ self.handle() 2020-01-21 00:41:32,246 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 426, in handle self.handle_one_request() 2020-01-21 00:41:32,247 [Thread-1 ] [ERROR] File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request method() 2020-01-21 00:41:32,248 [Thread-1 ] [ERROR] File "./weatherstation.py", line 751, in do_POST process_post_data(self.path, post_data) 2020-01-21 00:41:32,249 [Thread-1 ] [ERROR] NameError: name 'process_post_data' is not defined 2020-01-21 00:41:32,249 [Thread-1 ] [ERROR] ---------------------------------------- 2020-01-21 00:43:40,168 [Thread-1 ] [INFO ] "GET / HTTP/1.1" 200 - 2020-01-21 00:43:40,374 [Thread-1 ] [INFO ] "GET /favicon.ico HTTP/1.1" 200 - Just to make sure I knew, issuing 'python -V' at the command line confirmed my RPi is running python 2.7.13. Is WeatherPoly written for python 3.7 by chance? Kinda looks like it. On the bright side, at least it looks like the data file arrived at the Polisy webserver as intended, and was recognized as a POST request.