Polyglot weather station node servers

[bpwwer]

30 minutes ago, glarsen said:

@bpwwer, I installed this NS  on my Polisy and other than having to manually group the forecast periods in the AC, it all looks good using the data from my PWS.  

I am curious though about the et0 numbers in the forecasts. Are they acquired from Aeris, or are they calculated?  I'm interested to compare them to the numbers from both my Davis VP2+ and our local Govt Agricultural office (which sometimes align, and often don't).

Also, what is the "Climate Coverage" driver ?  It's blank in all forecasts and local data.

Thanks!

The et0 value is calculated.

The three values; coverage, intensity, and condition make up the current condition string.  Coverage would be strings like "areas of" or "likely" or "periods of".  Blank just means that there's no qualification on the condition.   You'd read these like:   "periods of" "light" "rain".  In the data, these are sent in a coded format and I've noticed that the coverage component does tend to be blank a lot.

[glarsen]

Again, ok, thanks Bob. I don't use that field for obvious reasons in the Rainmachine Parser and had forgotten what it was.

[glarsen]

@bpwwer, I notice that there is a marked difference in the et0 calculations between DarkSky and Aeris. For example:

The Aeris one is more consistent with current local forecasts.  Though the conditions reported by both are similar, I would have thought with a wind forecast as high as  shown by Aeris, their numbers would be higher, and DarkSky doesn't even forecast wind.  Any thoughts?

Both have the same elevation and plant type set in parameters.

Thanks!

Edited March 20, 2020 by glarsen

[bpwwer]

1 hour ago, glarsen said:

@bpwwer, I notice that there is a marked difference in the et0 calculations between DarkSky and Aeris. For example:

The Aeris one is more consistent with current local forecasts.  Though the conditions reported by both are similar, I would have thought with a wind forecast as high as  shown by Aeris, their numbers would be higher, and DarkSky doesn't even forecast wind.  Any thoughts?

Both have the same elevation and plant type set in parameters.

Thanks!

They are both using the same calculation so any difference is based only on the different inputs.  The main inputs are temperature, humidity, wind speed, solar radiation, elevation, and plant type coefficient. 

[glarsen]

I suspected as much. It's surprising that the input data varies so much for the same coordinates but I guess that's the nature of the beast. I'll watch them both for a while and just pick the one that aligns the best with local data.  Thanks.

[JeffE]

On 5/25/2019 at 11:19 PM, Bumbershoot said:

The ISY Dashboard doesn't show any data for DarkSky, as it appears to be a simple ON/OFF switch.  The ISY Admin Console, on the other hand, displays plenty of data from DarkSky, and the nodes are fully available in ISY programs.  See screenshots.

Did this ever get resolved?

[DennisC]

Dashboard wasn't designed for Node Servers.

[bpwwer]

The Dashboard author would have to add code to support each node server.

[simplextech]

35 minutes ago, bpwwer said:

The Dashboard author would have to add code to support each node server.

Dashboard author... you mean UDI.

[Michel Kohanim]

@Dashboard is history!

With kind regards,

Michel

[cosyn]

@bpwwer and @Michel Kohanim, I'm seeing strange behavior in rain data handoff from weatherpoly to ISY. 

Here's my set up: Polisy with weatherpoly set up to obtain data from my Davis Vantage Pro2 via CumulusMx. The data appear in the ISY correctly: in the WeatherPoly in the ISY main tab, precipitation shows as 0 inches for the past day and for today. Yep, we're having a drought in the Bay Area. 

I have a program that looks for the amount of rain today or yesterday and kicks off another set of programs. If it's rained >0.2 inches, I don't want the irrigation to turn on. If it hasn't rained more than this, I want the irrigation to come on. I have different irrigation valves that turn on depending on whether it's rained a certain amount over different stretches of time.

The system however reads that any amount of rain below 0.5" as having rained. See the program in the screenshot enclosed. So with a cutoff set anywhere <0.5" of rain (screenshot shows 0.3"), the ISY interprets incorrectly that 0" of rain has having rained. If I raise the cutoff to 0.5" or above, the ISY interprets correctly with 0" of rain that it has indeed not rained.

I had this same program set up with the Climate module and had no issues for many years. Thoughts about what might be happening?

Thanks in advance.

[madcodger]

Your program actually says ">=0.3". Have you tried ">0.2"? My theory is that because the VP2 measures to 0.01", you might have a rounding problem.

Absent that being the issue, it may be that the rounding of either the ISY is not sufficiently granular for this, or perhaps the rounding in Bob's nodeserver. The fact that 0.5" lappears to be the "cutoff" seems to suggest that. 

Another approach might be to try setting up a state variable, in which ">0.2 inches" sets the variable to 1 (on) while "<=0.2" turns it off (sets it to 0). It would be interesting to see if things are processed differently for variables as the trigger.

Just a couple of thoughts to explore on the path ro a solution.

[bpwwer]

It does seem like a rounding issue but almost like it's the constant that's being rounded.  I.E. your if is behaving as if 0.3 is being rounded to 0.0 and 0.5 then is rounded to 1.0. 

This node server in particular is difficult to debug because all of the node definitions it creates are dynamically created based on your configuration.   Thus it's really hard for someone else to try and reproduce what you're seeing. 

You might want to experiment with a simple program and also check the node server's log and verify the values being sent to the ISY for the daily/yesterday rain.

[cosyn]

On 3/24/2020 at 2:21 AM, madcodger said:

Your program actually says ">=0.3". Have you tried ">0.2"? My theory is that because the VP2 measures to 0.01", you might have a rounding problem.

Absent that being the issue, it may be that the rounding of either the ISY is not sufficiently granular for this, or perhaps the rounding in Bob's nodeserver. The fact that 0.5" lappears to be the "cutoff" seems to suggest that. 

Thanks @madcodger for the troubleshooting tips. I changed the >= to a > and that worked. It no longer interprets the statement as being true for values less than what is indicated.

Luckily there was 0.12" of rain yesterday and 0.04" today to troubleshoot ( @bpwwer I confirmed this in the node server log and ISY Main screen for the Precipitation node):

     - I started by changing the operator to > instead of >=.

     - Setting a threshold of >0.2" for Rainfall Yesterday, the statement was read as false.

          - Same when setting a value of 0.3 or 0.4".

          - Setting a value of >0.1", the statement was read as true.

     - For Daily Rainfall, setting a threshold of >0.05" the statement was read as false.

          - It was read as true when the value was reduced to 0.03".

It seems there's a rounding up issue with the >= operator.

Thanks

Edited March 25, 2020 by cosyn
formatting and value edit

[madcodger]

Great! Glad that worked. Happy watering.

[bluerogueman]

Hi Bob,

Polyglot complains about having insufficient date from OpenWeatherMap. I suspect there's a coding error somewhere.

 

It's now Wednesday, but Forecast 0 wrongly equals to Sunday and Forecast 1 and 2 correctly equal to Thursday and Friday.

 

Any ideas?

Patrick

 

[bpwwer]

13 hours ago, bluerogueman said:

Hi Bob,

Polyglot complains about having insufficient date from OpenWeatherMap. I suspect there's a coding error somewhere.

It's now Wednesday, but Forecast 0 wrongly equals to Sunday and Forecast 1 and 2 correctly equal to Thursday and Friday.

Any ideas?

Patrick

That's expected/intended behavior because of the way OpenWeatherMap provides forecast data.  They don't provide daily forecasts, but rather 3hour blocks. I combine the 3hr blocks into a daily forecast but at certain points in the day, they'll only provide a subset of 3hr blocks and not enough data to create a daily forecast.  When that happens, the node server provides the above notice. 

My thought was that providing no information was better than taking a forecast for 9pm - 12pm and letting you think it was a daily forecast with just really weird, useless values.

Creating separate nodes for each 3hr forecast didn't seem like the right thing to do either as that would mean you'd have about 40 forecast nodes and no way to visualize or create programs that deal with daily information. 

[gviliunas]

Re. Forecast Data from Open Weather Map

Hi Bob,

I am experiencing the same issue as Patrick regarding the Forecast 0 data presented. In my case, it is Sunday, Forecast 0 is reporting as Tuesday (From last Tuesday?), and Forecast 1 and 2 are reporting as Monday and Tuesday. Now, Alexa is telling me to expect a high temperature of 14 degrees C but Forecast 0 is only predicting 5.2 degrees C.

My question is, "How much can I trust the current day forecast, (Forecast 0)?" Is any of the data correct on this page or has none of it been updated since last Tuesday? If none of it has been updated since last Tuesday, maybe it would be better to break-out Forecast 0 into eight separate 3hr forecasts. Maybe a "x-minutes since last updated field" would be good to show on the forecasts pages as well.

Currently, I think my only other option if I want a forecast for today, is to store the next days forecast values (Forecast 1) and then tomorrow compare the day field in Forecast 0 with the current day. If these are not equal then not enough data was received and I should use the 24-hr old data from yesterday's Forecast 1. This is also not ideal but probably better than using a days-old forecast.

See attached screen shots

[bpwwer]

On 4/26/2020 at 8:25 AM, gviliunas said:

Re. Forecast Data from Open Weather Map

Hi Bob,

I am experiencing the same issue as Patrick regarding the Forecast 0 data presented. In my case, it is Sunday, Forecast 0 is reporting as Tuesday (From last Tuesday?), and Forecast 1 and 2 are reporting as Monday and Tuesday. Now, Alexa is telling me to expect a high temperature of 14 degrees C but Forecast 0 is only predicting 5.2 degrees C.

My question is, "How much can I trust the current day forecast, (Forecast 0)?" Is any of the data correct on this page or has none of it been updated since last Tuesday? If none of it has been updated since last Tuesday, maybe it would be better to break-out Forecast 0 into eight separate 3hr forecasts. Maybe a "x-minutes since last updated field" would be good to show on the forecasts pages as well.

Currently, I think my only other option if I want a forecast for today, is to store the next days forecast values (Forecast 1) and then tomorrow compare the day field in Forecast 0 with the current day. If these are not equal then not enough data was received and I should use the 24-hr old data from yesterday's Forecast 1. This is also not ideal but probably better than using a days-old forecast.

See attached screen shots

Are you getting the notice of insufficient data?  If not, then this is a different issue.  When that notice is displayed, no data is being sent to the ISY for forecast_0, otherwise it is.

This happens when OpenWeatherMap stops sending complete data for the current day's forecast. Prior to this point in time, the data sent to the ISY should be the correct data for the day so you should always have the most recent full current day forecast in forecast_0. It should switch over when the day switches over.

There is one exception to this and that's when the node server is started and on the initial query it doesn't get enough data to make a current day's forecast.  Then you'll get a empty or blank forecast as shown in bluerogueman's post above.

What you're showing above looks more like it hit some corner case where the data from OpenWeatherMap isn't complete or didn't match what was expected.   I don't know that I can really do anything to make OpenWeatherMap data presentable to the ISY.  It's just not very compatible with the ISY/NS display format.

[gviliunas]

Hi Bob,

Thank you for your response. The plot thickens....  I believe my problem has to do with the current day forecast (Forecast 0).

It is Monday and yet I noticed that Forecast 0 is still saying Sunday.

Okay, I deleted the NS completely and then re-added it back to the same slot. Immediately after doing this, I noticed all five forecasts were populated with zero data and all stated Sunday. I expect that this is correct.

After a few minutes, Forecasts 1-4 all updated to reasonable days and values but Forecast 0 still contained zero values and was still saying it was Sunday. I found the line below in the log:

                        DEBUG owm:query_forecast: Skipping update for forecast_0 because it lacks 8 records.

Maybe this has something to do with my location or the Open Weather Map server I am connecting to (Vancouver, Canada). I attached a log package in hope that this will shed some light on the issue.

Thanks for your help!!!!!

Greg

OpenWeatherMap_logs_2020-4-27_170119.zip

Edited April 28, 2020 by gviliunas

[bpwwer]

17 hours ago, gviliunas said:

Hi Bob,

Thank you for your response. The plot thickens....  I believe my problem has to do with the current day forecast (Forecast 0).

It is Monday and yet I noticed that Forecast 0 is still saying Sunday.

Okay, I deleted the NS completely and then re-added it back to the same slot. Immediately after doing this, I noticed all five forecasts were populated with zero data and all stated Sunday. I expect that this is correct.

After a few minutes, Forecasts 1-4 all updated to reasonable days and values but Forecast 0 still contained zero values and was still saying it was Sunday. I found the line below in the log:

                        DEBUG owm:query_forecast: Skipping update for forecast_0 because it lacks 8 records.

Maybe this has something to do with my location or the Open Weather Map server I am connecting to (Vancouver, Canada). I attached a log package in hope that this will shed some light on the issue.

Thanks for your help!!!!!

Greg

It sounds like it's all working as designed. 

OpenWeatherMap doesn't provide daily forecasts, it provides 3hr forecasts starting sometime around the time of the query out for 5 days.  So at the beginning of the day, it sends enough data to calculate a daily forecast, then at some point during the day it doesn't.  From that point on, the node server can't calculate a forecast for the current day because OpenWeatherMap isn't sending it.  Because you restarted the node server in the middle of the day, OpenWeatherMap is not sending enough data to generate a forecast for the current day, thus Forecast_0 is left blank (or all values are zero).

This is also complicated by the fact that OpenWeatherMap is sending the data based on universal time.  This means that you don't start getting a full days worth of forecast data for your current day at midnight your time.  Depending on your time zone it's going to be either sometime before midnight or sometime after midnight.

I believe I'm doing the best I can to get an accurate Forecast_0 out of OpenWeatherMap. Once the node server has enough data to calculate the daily forecast for the current day, it should be updating Forecast_0 with that data. It should then remain mostly constant for the rest of the day.

If you truly need a good current day forecast, you should look at one of the other services that provide daily forecasts directly.

[gviliunas]

Thanks for your reply Bob,

I just checked and all of the forecasts (0-4) now look reasonable. I don't know how Forecast-0 became stuck five days behind but I know how to check for this now. Thanks for your help in sorting this out and setting me straight.

[gviliunas]

Hi Bob,

After two days, my Forecast 0 was again having trouble obtaining a good set of data from Open Weather and was stuck several days behind. At that point, I took your advice and tried WeatherBit instead and this has been working perfectly! Maybe the data for my location is sporadic causing the problems with Open Weather data. In any event, this is fixed now.

Thank you for all your hard work writing and supporting these node servers!!

[DaveStLou]

On 2/23/2020 at 4:52 PM, bigDvette said:

Not sure if I missed instructions somewhere, but finally figured out I had to map the nodes form this list:  https://github.com/bpaauwe/WeatherPoly to positions in the url.  I can't see the url coming in so I looked it up in CRT.py.  

I added your changes to my CRT.py file that I use to publish data for a local weather page using the realtime.txt.  

I can create the variables for yesterday records you have commented out if anyone would like them.  I'm mostly writing this to document what I had to do to get it working.  If mapping temperature-main = 2 was incorrect then I'd like to know the right method.  Also, in the CRT.py temp is line 3 but the first 2 or dates so trail and error got me there.

I have a request.  I actually pull in some other weather data through some extensions to weewx. In particular I pull in the lake level from the geological site.

Could we add node category and maybe call it environment.  I'd like variables for lakelevel (current, max_year, min_year, max_month, min_month, min_day, max_day) . It could just be water level so it is more generic.  All these are in feet of elevation so convert from meter to feet ...

I could see other variables like snow in this category.  If I still had polyglot on my raspberry instead of the polisy I'd hack the files to see I could get it working. 

I know this is going back awhile, but my CumulusMX blew up this week speeding up my move to WeeWx - something I've been considering for sometime. I've read through the discussion and think I understand it. I'm getting some errors on the WeeWx side after installing the extension, which appear to be related to running under Python 3.

@bigDvette, if you wouldn't mind sharing the variable information, that would help after I get the Python errors resolved

[bgrubb1]

I am helping a friend install WeatherflowPoly on his new Polisy (runs fine on mine:-) )

everything seems to start fine until I get the following error

 

I believe WeatherflowPoly is attempting to hit the weatherflow server and doesn't like his station ID

I have confirmed the station ID in the app and through their website

Am I missing anything obvious ??

Thanks in advance

...Barry