Polyglot weather station node servers

[bpwwer]

I've created a number of node server to access data from personal weather stations.  They are all of the same basic design and share a lot of similarities.  This topic can be used to report issues, requests features, and request new node servers for other weather stations.

Currently available

• WeatherFlowPoly - Works with WeatherFlow (www.weatherflow.com) smart weather stations, pulling local data from the station only.
  • WeatherFlowPGC - Similar to above, but gets it's data from the WeatherFlow servers which allows this to run from Polyglot Cloud
• AmbientPoly - Works with the Ambient Weather Network. Data comes from the Ambient Weather servers. Works with both local and cloud based Polyglot
• MeteoBridge - Works with a MeteoBridge (www.meteobridge.com) weather bridge. Live data is queried from the local MeteoBridge device. This should also work with Ambient's WeatherBridge. Works only with a local Polyglot
• WeatherPoly -  Import weather data from local weather software. Currently supports MeteoBridge, Cumulus, and AcuParse.  Has the potential to support any software that can export weather data to a web URL.  Works only with a local Polyglot
• DavisWeather - Works with the Davis WeatherLinkIP, the Serial/USB Data Logger with the WeatherLink Computer Software, or the Vantage Connect weather station solutions.
• udi-darksky-poly - Pulls weather data from the DarkSky weather service. Currently pulls current conditions and the 7 day forecast. Works with both local and cloud based Polyglot.
• udi-owm-poly - Pulls weather data from the OpenWeatherMaps weather service. Pulls both current conditions and forecasts. Works with both local and cloud based Polyglot
• udi-apixu-poly - Pulls weather data from the APIXU weather service. Pulls both current conditions and forecasts. Works with both local and cloud based Polyglot (no longer available)
• udi-weatherbit-poly - Pulls weather data from the weatherbit.io service. Pulls both current conditions and forecast.
• udi-aeris-poly - Pulls weather data from the AERIS weather service.  Pulls both current conditions and forecast. Can be configured to get current conditions from a PWS station. Note that AERIS does not have a free plan available except for folks that are contributing data via PWSWeather.com.
• udi-climacell-poly - Pulls weather data from the Climacell weather service. Pulls both current conditions and forecasts. See www.climacell.co/pricing for subscription plans.

In development

• wdpoly - Works with Weather Display (www.weather-display.com) weather software. Data is collected locally, directly from the software.

Notes

• The udi-darksky-poly, udi-owm-poly, udi-weatherbit-poly, udi-climacell-poly, and udi-aeris-poly are all very similar.  They differ slightly in the specific weather values each supports and number of days of forecast data available. There should be no reason to run more than one of these. Pick the one that best meets your needs.

I'm open to creating new node servers for other weather stations. But without direct access to the station (or at least the station's data), I'd need to rely on someone with the station to test and debug.

[DaveStLou]

8 hours ago, bpwwer said:

I'm open to creating new node servers for other weather stations. But without direct access to the station (or at least the station's data), I'd need to rely on someone with the station to test and debug.

I have a Davis Vantage Pro2 weather station running Cumulus on a Raspberry Pi. I would be interested in assisting in developing a node server. The Cumulus Wiki can be found here: http://wiki.sandaysoft.com

Edited September 27, 2018 by DaveStLou

[bpwwer]

17 hours ago, DaveStLou said:

I have a Davis Vantage Pro2 weather station running Cumulus on a Raspberry Pi. I would be interested in assisting in developing a node server. The Cumulus Wiki can be found here: http://wiki.sandaysoft.com

I took a quick look at Cumulus and I didn't see anything about pushing/pulling data over the local network.  So far, all the node servers use a network connection to the station (or weather software) to access the data that they forward on to the ISY.  If Cumulus has the ability to either make a socket connector or output the data via a UDP broadcast, it would be fairly easy to support.

There are a couple of other options for creating a node server if it doesn't support a socket/UDP method.  

1) It does generate a file; reatime.txt. A node server could read that for data. This would require that the node server have access to the file.

2) Run a simple server on the node server that listens for updates from Cumulus. This assumes that Cumulus can be configured to send data to a user-defined weather service

Weewx software is similar to Cumulus and either of the two solutions would work for it.

#2 seems like a more generic solution but is more effort.   I'll start looking into it as I have time.

[DaveStLou]

8 minutes ago, bpwwer said:

I took a quick look at Cumulus and I didn't see anything about pushing/pulling data over the local network.  So far, all the node servers use a network connection to the station (or weather software) to access the data that they forward on to the ISY.  If Cumulus has the ability to either make a socket connector or output the data via a UDP broadcast, it would be fairly easy to support.

There are a couple of other options for creating a node server if it doesn't support a socket/UDP method.  

1) It does generate a file; reatime.txt. A node server could read that for data. This would require that the node server have access to the file.

2) Run a simple server on the node server that listens for updates from Cumulus. This assumes that Cumulus can be configured to send data to a user-defined weather service

Weewx software is similar to Cumulus and either of the two solutions would work for it.

#2 seems like a more generic solution but is more effort.   I'll start looking into it as I have time.

I don't believe it has any option for socket connector or UDP broadcast. I searched their forum: http://sandaysoft.com/forum

1) It saves the realtime.txt locally. In my case, I have mounted my NAS and instead send it there. So mine could also be visible to the RPi that is running the Polyglot on the same LAN. Let me know if you'd like me to PM an example of the realtime.txt file to you for review.

2) Cumulus can be configured to send data to a user-defined service similar to the Weewx.

Below are a couple of screenshots of a portion the configuration called "Internet Settings"

[bpwwer]

I have a new node server available for testing.  It's called WeatherPoly and it is designed to accept weather data from various weather software packages. I've been testing it a bit with both MeteoBridge and Cumulus, but in theory, it should be able to support any weather software that can send data via an HTTP GET request to a user defined server.

It's not yet in the node server store so a manual installation is necessary.  To manually install:

1. git clone git@github.com:bpaauwe/WeatherPoly.git to your .polyglot/nodeserver directory. 
2. Locate it in the local node servers list from Polyglot's add node server screen and add it.

To make this work requires configuration of both the weather software and the node server so that a mapping can be created between the data sent by the weather software and the nodes created by the node server.

Using Cumulus as an example:

1. Create a Custom HTTP entry under Internet settings.
2. Use the polyglot machine's IP and the port number defined in the node server (default is 8080) 
3. Create a query string with all the information you want to send (see the Cumulus webtags reference for what data can be sent)
4. You will need to specify names for each data value you send
5. Set how often to send the data (like 60 seconds) 

You should end up with something that looks like:

http://192.168.1.12:8080/cumulus?temperature=<#temp>&humidity=<#hum>&pressure=<#press>

In the node server configuration, you'll need to map the temperature, humidity, and pressure listed above to nodes.  The README.md has a full list of the nodes the values each node supports. To map the above three values you'd likely use:

temperature-main  | temperature
humidity-main     | humidity
pressure-station  | pressure

 

[DaveStLou]

8 hours ago, bpwwer said:

I have a new node server available for testing.  It's called WeatherPoly and it is designed to accept weather data from various weather software packages. I've been testing it a bit with both MeteoBridge and Cumulus, but in theory, it should be able to support any weather software that can send data via an HTTP GET request to a user defined server.

It's not yet in the node server store so a manual installation is necessary.  To manually install:

1. git clone git@github.com:bpaauwe/WeatherPoly.git to your .polyglot/nodeserver directory. 
2. Locate it in the local node servers list from Polyglot's add node server screen and add it.

I'm in the middle of a data center move but I'll try to get this going for testing as soon possible.

Step one, I'm running into this when I attempt to clone:

pi@pi1:~/.polyglot/nodeservers $ sudo git clone git@github.com:bpaauwe/WeatherPoly.git
Cloning into 'WeatherPoly'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
 

 

[bpwwer]

6 hours ago, DaveStLou said:

I'm in the middle of a data center move but I'll try to get this going for testing as soon possible.

Step one, I'm running into this when I attempt to clone:

pi@pi1:~/.polyglot/nodeservers $ sudo git clone git@github.com:bpaauwe/WeatherPoly.git
Cloning into 'WeatherPoly'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

Hmm, the repository is public so there shouldn't be any access rights issues.  I copied the command right off the github webpage.  

You can also try git clone https://github.com/bpaauwe/WeatherPoly.git

I don't think you need the sudo to execute the git clone, it should be fine running it with normal user permissions. 

[DaveStLou]

I just downloaded it and got it going. To keep things simple to start with, I set up only temperature, humidity and pressure as you outlined. 

Nothing showing up in ISY other than it appears in a slot under the Configure. It is not appear as a Node Server in the list.

This is what is showing in the WeatherPoly log.

2018-10-10 17:54:00,901 INFO     Configuration Change...
2018-10-10 17:54:00,902 INFO     Found difference with saved configuration.
2018-10-10 17:54:00,902 INFO     Check for existing configuration value
2018-10-10 17:54:00,903 INFO     Trying to create a mapping
2018-10-10 17:54:00,903 INFO     MAPPING humidity to humidity-main
2018-10-10 17:54:00,903 INFO     MAPPING pressure to pressure-station
2018-10-10 17:54:00,904 ERROR    KeyError in gotConfig: 'station\ufeff\ufeff'
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.5/site-packages/polyinterface/polyinterface.py", line 459, in inConfig
    watcher(config)
  File "./weatherstation.py", line 53, in process_config
    self.map_nodes(config)
  File "./weatherstation.py", line 317, in map_nodes
    'driver': write_profile.PRES_DRVS[vmap[1]],
KeyError: 'station\ufeff\ufeff'

Let me know if you want me to PM the log package

Edited October 11, 2018 by DaveStLou

[bpwwer]

25 minutes ago, DaveStLou said:

I just downloaded it and got it going. To keep things simple to start with, I set up only temperature, humidity and pressure as you outlined. 

Nothing showing up in ISY. This is what is showing in the WeatherPoly log.

2018-10-10 17:54:00,903 INFO     MAPPING pressure to pressure-station
2018-10-10 17:54:00,904 ERROR    KeyError in gotConfig: 'station\ufeff\ufeff'
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.5/site-packages/polyinterface/polyinterface.py", line 459, in inConfig
    watcher(config)
  File "./weatherstation.py", line 53, in process_config
    self.map_nodes(config)
  File "./weatherstation.py", line 317, in map_nodes
    'driver': write_profile.PRES_DRVS[vmap[1]],
KeyError: 'station\ufeff\ufeff'

Let me know if you want me to PM the log package

Looks like there may be some  extra unicode characters at the end of station in the pressure-station key, try deleting and re-adding that configuration.

[DaveStLou]

3 minutes ago, bpwwer said:

Looks like there may be some  extra unicode characters at the end of station in the pressure-station key, try deleting and re-adding that configuration.

That was it. (Copying and pasting from the instructions isn't always a good idea...)

Now it's hung here:

2018-10-10 19:06:43,753 INFO     Configuration Change...
2018-10-10 19:06:43,754 INFO     Found difference with saved configuration.
2018-10-10 19:06:43,754 INFO     Check for existing configuration value
2018-10-10 19:06:43,755 INFO     Trying to create a mapping
2018-10-10 19:06:43,755 INFO     MAPPING temperature to temperature-main
2018-10-10 19:06:43,755 INFO     MAPPING pressure to pressure-station
2018-10-10 19:06:43,756 INFO     MAPPING humidity to humidity-main
2018-10-10 19:06:43,756 INFO     Try to create node definition profile based on config.
2018-10-10 19:06:43,757 INFO     write_profile: Writing profile/nodedef/nodedefs.xml

 

[DaveStLou]

I don't see that file in the folder. I wonder if i have a permission issue. Looking into it.

Edited October 11, 2018 by DaveStLou

[DaveStLou]

I had to create the nodedef subfolder and then it was able to write the file.

The nodes are now showing up but there are no values showing in ISY but they do appear in the Polyglot Node Details view.

[bpwwer]

You may have to restart the admin console to get it to properly read the new nodes that were installed.

I'll take a look at the problem with writing the nodedef.  The directory should be part of the git repository, but because it was empty, it may not have been included.

 

[DaveStLou]

28 minutes ago, bpwwer said:

You may have to restart the admin console to get it to properly read the new nodes that were installed.

I've restarted the Admin Console and even tried rebooting the ISY but no values reflected on that side.

[bpwwer]

14 hours ago, DaveStLou said:

I've restarted the Admin Console and even tried rebooting the ISY but no values reflected on that side.

That's strange.  I'm not able to reproduce this.  The way it's supposed to work is that the node server sends the updates to Polyglot and Polyglot sends the updates to the ISY. Since you're seeing the updates in the Polyglot frontend, that implies that the node server is working correctly.

My only suggestion would be too delete and re-add the node server.  That should clear out Polyglot's database .

I was going to push a new version to git hub, but it looks like my RPi just died and I won't be home to reboot it until this evening.

[DaveStLou]

On 10/11/2018 at 10:42 AM, bpwwer said:

That's strange.  I'm not able to reproduce this.  The way it's supposed to work is that the node server sends the updates to Polyglot and Polyglot sends the updates to the ISY. Since you're seeing the updates in the Polyglot frontend, that implies that the node server is working correctly.

My only suggestion would be too delete and re-add the node server.  That should clear out Polyglot's database .

I was going to push a new version to git hub, but it looks like my RPi just died and I won't be home to reboot it until this evening.

Okay. I deleted the node server and deleted the folder and started over. Unzipped the download file again, recreated the node server. This time I'm seeing this in the Polyglot log:

2018-10-12 20:13:00 - debug: MQTTP: nsCommand Message: {"status":{"value":"30.042","address":"pressure","uom":117,"driver":"ST"},"node":"11"}
2018-10-12 20:13:00 - debug: MQTTP: nsCommand Message: {"status":{"value":"30.042","address":"pressure","uom":117,"driver":"ST"},"node":"11"}
2018-10-12 20:13:00 - info: WeatherPoly(11): Processing command: status
2018-10-12 20:13:00 - info: WeatherPoly(11): Processing command: status
2018-10-12 20:13:00 - debug: ISY: 404 - https://10.42.8.81:443/rest/ns/11/nodes/n011_temperature/report/status/ST/44.9/4
2018-10-12 20:13:00 - debug: ISY: 404 - https://10.42.8.81:443/rest/ns/11/nodes/n011_temperature/report/status/ST/44.9/4
2018-10-12 20:13:00 - error: WeatherPoly(11): n011_temperature or ST does not exist - ISY returned 404
2018-10-12 20:13:00 - error: WeatherPoly(11): n011_temperature or ST does not exist - ISY returned 404
2018-10-12 20:13:01 - debug: ISY: 404 - https://10.42.8.81:443/rest/ns/11/nodes/n011_pressure/report/status/ST/30.042/117
2018-10-12 20:13:01 - debug: ISY: 404 - https://10.42.8.81:443/rest/ns/11/nodes/n011_pressure/report/status/ST/30.042/117
2018-10-12 20:13:01 - error: WeatherPoly(11): n011_pressure or ST does not exist - ISY returned 404
2018-10-12 20:13:01 - error: WeatherPoly(11): n011_pressure or ST does not exist - ISY returned 404

I can see the WeatherPoly node server in ISY but there are no nodes showing in the device lists. I've tried adding all nodes but no change.

[DaveStLou]

Progress! I started over entirely.  I got the git clone to work and installed it from command line on the Rpi.

The node server installed and I configured temperature, humidity and pressure as well as changed it to "us" from "metric".

On the ISY side the node appeared under the Node Servers on the top menu bar but never created the nodes under devices - nor did it create them when I requested it to "Add All Nodes"

So I manually created them using the node addresses from the Nodes tab in the Polyglot dashboard.

Now the three nodes are showing but the values are incorrect. Here's an example:

2018-10-13 07:56:00,028 DEBUG    Set temperature driver ST to ['43.4']
2018-10-13 07:56:00,029 INFO     Updating Driver temperature - ST: 110.1, uom: 17

The actual temperature being sent from Cumulus is 43.4F which appears to be being received above by WeatherPoly but the temperatures node in ISY shows as 110.1. So either I have something backwards in my configuration or something's wrong in the node server. Humidity is showing correct but pressure is inverted similar to temperature.

Like I said, progress!

Here's the rest from the log for context:

2018-10-13 07:56:00,024 DEBUG    Set pressure driver ST to ['30.119']
2018-10-13 07:56:00,026 INFO     Updating Driver pressure - ST: 0.889, uom: 23
2018-10-13 07:56:00,028 DEBUG    Set temperature driver ST to ['43.4']
2018-10-13 07:56:00,029 INFO     Updating Driver temperature - ST: 110.1, uom: 17
2018-10-13 07:56:00,031 DEBUG    Set humidity driver ST to ['94']
2018-10-13 07:56:00,033 INFO     Updating Driver humidity - ST: 94.0, uom: 51

 

Edited October 13, 2018 by DaveStLou

[DaveStLou]

27 minutes ago, DaveStLou said:

The actual temperature being sent from Cumulus is 43.4F which appears to be being received above by WeatherPoly but the temperatures node in ISY shows as 110.1. So either I have something backwards in my configuration or something's wrong in the node server. Humidity is showing correct but pressure is inverted similar to temperature.

 

I realized after I posted that the temperature and pressure were still showing in metric even though I had changed it. I deleted units line in the config and entered it new again as us and restarted. Now reporting correctly in ISY

[bpwwer]

1 hour ago, DaveStLou said:

I realized after I posted that the temperature and pressure were still showing in metric even though I had changed it. I deleted units line in the config and entered it new again as us and restarted. Now reporting correctly in ISY

I haven't figured out how I want to handle the units yet.  For all the other weather node servers I've created, the incoming data units are fixed (typically metric) so configuring the display units in the node server is just a simple conversion.  But for this one, the incoming data could be in any units.  Right now, it is still assuming that the incoming units are metric so when you changed it to US, it applied a metric-to-us conversion to the temperature.  I'm not sure why that's not still happening though.

I think what I'm going to end up doing is let you specify the incoming and display units separately.  That makes thinks a bit more complicated as I then have to do multiple types of conversions vs. the simple one way conversions I do now.

 I'll look in to trying to re-install from scratch, but I I suspect the problem you saw may be related to a known bug in Polyglot where it doesn't clean up the database completely when a node server is deleted.

[DaveStLou]

19 hours ago, bpwwer said:

I haven't figured out how I want to handle the units yet.  For all the other weather node servers I've created, the incoming data units are fixed (typically metric) so configuring the display units in the node server is just a simple conversion.  But for this one, the incoming data could be in any units.  Right now, it is still assuming that the incoming units are metric so when you changed it to US, it applied a metric-to-us conversion to the temperature.  I'm not sure why that's not still happening though.

I think what I'm going to end up doing is let you specify the incoming and display units separately.  That makes thinks a bit more complicated as I then have to do multiple types of conversions vs. the simple one way conversions I do now.

I'll look in to trying to re-install from scratch, but I I suspect the problem you saw may be related to a known bug in Polyglot where it doesn't clean up the database completely when a node server is deleted.

I think it is the Polygot database bug which I've run into before.

Regarding the unit type, I was so focused on the values, I hadn't notice it was displaying incorrectly. I looked and there's no option for designating the unit in the Cumulus webtag. It's set system wide:

[DaveStLou]

If it's helpful, Cumulus CAN send the unit types being used separately in these webtags

Edited October 14, 2018 by DaveStLou

[DaveStLou]

Another thought regarding units - Could the logic be backwards? It seems like when "metric" is selected as units in WeatherPoly the values displayed in ISY are actually "us" degrees but it displays F behind the degree mark. But when "us" is selected as units in WeatherPoly the values are converted to celsius and displayed in ISY with a C behind the degree mark. 

[tlightne]

I am running an Acurite 5n1 personal weather station with  their new hub sharing my data with Wunderground. Is there a node server for my weather station? 

[bpwwer]

18 hours ago, DaveStLou said:

Another thought regarding units - Could the logic be backwards? It seems like when "metric" is selected as units in WeatherPoly the values displayed in ISY are actually "us" degrees but it displays F behind the degree mark. But when "us" is selected as units in WeatherPoly the values are converted to celsius and displayed in ISY with a C behind the degree mark. 

The unit conversion (metric -> us/uk) is there since I used one of my existing node servers with working unit support as the basis for creating this one, but I haven't looked at any of the code that sets the ISY "unit of measure" so I don't know if any of that carried over. But in any case, it only ever had the ability to do the one way conversion.

I think I'll add manual settings for incoming and display units to start, that should at least get things working. Then I'll take a look at using the webtag values to get the incoming units.

[bpwwer]

15 hours ago, tlightne said:

I am running an Acurite 5n1 personal weather station with  their new hub sharing my data with Wunderground. Is there a node server for my weather station? 

No, there isn't one for the Acurite.

I also have a 5n1 (actually a couple of them) and the changes they made with the latest Access hub is one of the main reasons I purchased a WeatherFlow station.  The Access hub encrypts the data it sends to the Acurite servers so it's more difficult to intercept and they don't provide any API access to the data on their servers, at least as far as I know.

There are a couple of ways to get access to the data

1. Install something like https://github.com/acuparse/acuparse
2. Create your own hub using https://github.com/merbanan/rtl_433 and a  Realtek RTL2832 based DVB dongle 

Both of those solutions require a fair amount of setup to get working.

I'm currently using method #2 and wrote a program to convert the output of rtl_433 into the format that WeatherFlow generates.  I used this to initially create the WeatherFlow node server.

I haven't looked at method #1 close enough to understand what it would take to create a node server that works with it, but if you do want to set up one of these methods or find some other way to access the data, let me know.