All time Rain data and NC historical Rain data

[smileyw]

Any chance we can get all time rain data added?  It would just be the same thing as the others, but you are just an epoch date of 0 (or whatever the equivalent of that is.)

Also, my tempest doesn't do the best job of getting rain data, so I really have the relay on the NC data instead.  Could I also suggest a variable that would allow me to use the NC data instead of observed?  What's odd about this one is that when I look at the API itself it seems to ONLY have NC data, but my totals are much lower than what WF's app is giving me (i.e. monthly in the node server is 3.93, but 5.09 in the app (which shows it is NC data.)

[bpwwer]

The behavior somewhat depends on how you've configured the node server.

When the node server starts, it queries the servers for the historical rain. I don't believe they provide an option to get total rain since the station came on line.  It is probably possible to generate queries to get that but I'm struggling to understand how that information would be valuable for automation.

If you've configured the node server to use local values from the hub, then historical rain ends up being what was queried initially plus what the hub reports locally.   The whole point of using the local data is to not have to rely on the internet connection for the data.  To have most data be local but rain be queried from the server breaks that design goal.

If you've configured the node server to use remote values from the server, then you're getting NC rain data from the servers and the values should be very close, if not exactly the same as what the app reports.

Unfortunately, WeatherFlow has decided to use server side processing to modify the values generated by the local sensors rather than fix the firmware to make the local sensors report correct data.  Which is why you have the choice, you can use the local sensor data and deal with any inaccuracies or use the server data and get the "corrected" data.  Your choice.

[smileyw]

okay, I should have thought about that.  That will probably fix half of it.  I will switch it to remote, and I suspect you are correct, and it will give me the NC data.

As for getting all time data, it would be useful for me specifically as I export that data to a dashboard that I display.  But maybe not as much use to others.  But it is a simple one to add, so hopefully someone will do it.  I could probably figure it out myself, but that may not be a good idea, my coding skills are not fantastic.

[smileyw]

oddly, switching it to remote didn't resolve showing the wrong data.  Now I am just confused.

[bpwwer]

Did you restart the node server after switching?

The node server isn't doing anything but reporting the data it gets from the server.

[smileyw]

yep.  I even deleted and re-added the node server.  Why would it have different data than the Weatherflow app?

[bpwwer]

That would be a question for WeatherFlow. I would think that their app gets the data the same way the node server does, but I have no way to confirm that.   When I first created the node server it matched what was reported on the app, but the app has changed significantly since then.

[smileyw]

On 3/25/2023 at 8:18 AM, bpwwer said:

That would be a question for WeatherFlow. I would think that their app gets the data the same way the node server does, but I have no way to confirm that.   When I first created the node server it matched what was reported on the app, but the app has changed significantly since then.

I just spent some time in the code in the NS, it is using the wrong data (or at least I would argue that it is...)

For the tempest (which is what I have), it is using 12 (Rain Accumulation (mm)) which it should be using 19 (NC Rain Accumulation (mm)).  That would give you a far more accurate answer.  My suggestion would be to have a config parameter to either use NC data (19) (as the App and everything else does) or use raw data (12) (which isn't very accurate for a sonic rain sensor).

I am not a coder, but it also looks like added "all time" would not be that hard.  It would mostly be adding it to the yearly and monthly subroutine.  If I knew more about how this is all structured, I would try to do it myself.

Edited March 26, 2023 by smileyw
added column numbers

[bpwwer]

The node server is using the official local API from WeatherFlow

https://weatherflow.github.io/Tempest/api/udp/v171/

According to that, there isn't a field 19.  Looks like they changed the remote API at some point to split the NC rain out and added new fields.  The node server is using the same code to process both local and remote as they're identical up through field 17.  So it's not as simple as just switching to field 19 as that will break the local handling.

I'll add support of the NC fields with remote to the todo list.

I didn't say it would be hard to add rain for all time, I just do see how that could be at all useful in a home automation setting.   Adding features that have no real value comes at a cost. It makes the code more complex and has to be maintained.  I'm not charging enough for the node server for me to want to do that.  The ROI for me, just isn't there.

[smileyw]

Thank you.  Looks like you are right.  They must have updated it.  I was using the swagger interface for my testing which is by default whatever the latest version is.

Here is where is why all time rainfall is useful...  If you are actually trying to display data, it is important.  In my case I am using a display to do it.  Frankly, why not just add whatever is realistically available?  I am using Home Assistant to aggregate everything and display on a Fire Tablet (or something).  I am now able to get the data from the NS into HA, so that's a big step forward. This is now the only thing that is missing for me to finish my project.  Will be used by others?  Maybe, maybe not, but it is fairly simple and should not hurt anything, and I would really appreciate it.

If I can help in some way, I am happy to do so. I don't code much at all, but I have been in tech for 30 years, and work in IAM security.

[bpwwer]

On 3/26/2023 at 10:12 AM, smileyw said:

Here is where is why all time rainfall is useful...  If you are actually trying to display data, it is important.  In my case I am using a display to do it. 

I still don't understand why displaying this data is important other than because you want to see it.  The node server (well all node servers) are used to get data into the ISY/IoX home automation controller.  Historical data, while interesting to view, isn't typically useful to enhance the control functions.  rainfall year to date could be of some use in controlling irrigation, but a three year total, not so much.

On 3/26/2023 at 10:12 AM, smileyw said:

Frankly, why not just add whatever is realistically available? 

Because it's not free.  It takes time to add features and more features means more complexity in the software and more complexity means more effort to maintain the software.  

On 3/26/2023 at 10:12 AM, smileyw said:

Will be used by others?  Maybe, maybe not, but it is fairly simple and should not hurt anything, and I would really appreciate it.

I've been providing a WeatherFlow module for a couple of different Home Automation systems for over 5 years and no one has asked for this before.  So I'm pretty confident that adding this will not increase sales at all.   So it really comes down to a business decision.  Is it worth my time to add this feature?  Or do I use that time for something else that will generate more income?  

While it may seem simple, it's not.  WeatherFlow doesn't make the data available via a database interface.  If we could just do something like "select * sum(rain);" it would be simple.  But the API can only query over specific time frames, "for all time isn't specific"  That means making queries using earlier and earlier time frames until it returns an error and accumulating all the records returned with each query.  Because the ISY/IoX isn't designed to maintain historical data, this process would happen every time the node server starts.  How much additional time it adds to the startup will depend on how long the station has been in operation, increasing over time.   Over time, because you're asking for all of station records, WeatherFlow may have to put limits on the queries.    So no, it's not simple, and there's a possibility of negatively impacting the node server performance.

[DennisC]

On 3/26/2023 at 1:12 PM, smileyw said:

Here is where is why all time rainfall is useful... 

You all ready have this information available to you in the Tempest app. If it is that important to you to have it in your automation (I agree with Bob, don't see the value of this in an automation system), then create a variable and add the total once a year. Then you can sum it year after year and display any way you would like.

[smileyw]

I guess some people use data, and some people ONLY do automation.  I do both.  As for your comment that others are not asking for it, that is not true, and least not with Home Assistant if you want to go looking around, in fact they have whole systems designed to store sensor data for just this reason.  with the EISY adding display and purposely marketing it for exactly what I am trying to do, I suspect there are (or will be) others in the same boat in the UD camp as well.  Think about the power data they are working hard to improve.  It is historical as well in the same way.  I think there is more than just automation here.  But I guess not everyone agrees with that.

@DennisCthat is exactly what I am planning on doing for now.  Hopefully Bob will change his mind, but I figure that isn't likely.  I could probably add the code myself, it is just a few lines over what is already there to have it done correctly (he is already making an API call and then parsing the data in the exact same way for month and year.  This one would just be a third one that sets sets the epochdate so some far back time, and do it a 3rd time.  i may try to figure out how to edit the code on the EISY directly to do that.  But I will need to wait until the updated data set is in so I don't have to do it twice as it is all going to change at that point.

 

[bpwwer]

48 minutes ago, smileyw said:

I guess some people use data, and some people ONLY do automation.  I do both.  As for your comment that others are not asking for it, that is not true, and least not with Home Assistant if you want to go looking around, in fact they have whole systems designed to store sensor data for just this reason. 

You still have not described how you will use the data.  The data is only valuable if it can be used in some way.  Displaying it so you can look at it and say "Wow, my system has recorded 4956 inches of rain since I started collecting data" isn't making use of the data.  The reason we collect data is to drive decisions with that data.   In general, for home automation, that means looking at real-time data and making decisions in real-time.  But that doesn't exclude making use of historic data. 

And I'll say it again, adding features is not free.  Since I can't see any way adding this would increase sales, I can work up an invoice on how much it will cost to add this for you if you'd like.

[smileyw]

Obviously we are going to have to agree to disagree and that’s okay.  For some reason your suggestion to “invoice me” really rubbed me the wrong way. I think it’s the formality of it. I’m not sure. It’s funny. I was thinking this morning about PM’ing you and ask you if I flipped you some money if you would do it, but I guess that’s not going to be an option.  anyway, if you read my last comment I already assumed  you don’t have any interest in doing anything that isn’t of immediate automation value. And that’s your prerogative. I do understand that and I see your point. I really do. 
But please update the data model to the current tempest API so at least I can at least have accurate data to make watering decisions (which of course is exactly what you want in an integration like this.). We can all agree the NC data is often far more accurate than observed data. Want proof?  The Rachio integration with WF only uses NC data. Having both would probably be best, but as you point out, it has costs.   

All of this is a good excuse to get back into scripting. I haven’t touched it since the VBS days. I guess it’s time to learn python.  If I do end up writing it I will give you what I write.  You can use it or not. That’s your call. I would just be a couple of lines in your existing subroutine.

[bpwwer]

I'm sorry that offering to provide a cost estimate rubbed you the wrong way.  But I create and sell node servers and part of my business and I have to make decisions based on how that will effect my business.  

If you were asking for something that I though would enhance the product I'd likely just do it, but despite me asking with every reply, you still have not shown how this would enhance my node server. 

It's not a question of how hard or easy it is to do or even a question of interest.  I certainly don't think I know best how data can be used or not.  I can be convinced.  Provide me with a real-world example of how reporting rain since system installation can be used to do something you can't do without it.  That's all I've been asking for and I don't think that's unreasonable.

There's no question that the NC data is valuable. I thought that's what was being returned in the server queries and didn't realize that they had added new fields for that until I looked it up after you asked about it.  WeatherFlow doesn't notify me of changes to the their API. Updating that is on my list and is fairly high on the list.

[Ross]

I completely respect @bpwwer's decision as to what he includes in the node server and what he doesn't.  It's his NS that he's selling/providing.  I use a few of his NS's and very much appreciate him and them and all the NS authors.  If we want something else, we'll need to learn how to create node servers ourselves (and I would like to do that with the SolarEdge API).

Regarding the reason for some data, I fall into @smileyw's camp.  I also display all kinds of info from my ISY and node servers.  Perhaps I don't exactly "do" anything with it; however, it does provide a smile to my face to know the info is right there on the display.  Some things that I displayed I eventually did do something with.  For instance, my soil temp was just a display until I got the idea to notify me when it got to a certain temp so we can plant certain things in our vegetable garden.

I have a very cool display I created in Google Sheets which pulls from various sources including ISY and NS info (weather, temps around the house-pool-soil, forecast, etc.) along with news and even events happening in the sky tonight.  My goal is to create a type of Magic Mirror.

Just my 2 cents that wasn't requested,

Ross

[bpwwer]

Thanks @Ross I appreciate the comments and thoughts.

I understand the desire to see every possible piece of data, but there has to be limits.  If we're going to add rain accumulation since the station came on-line shouldn't we then do that for everything?  Why not accumulated UV ratings since the station came on line.  Or accumulated temperatures, dewpoint, humidity, wind, etc.  All of these are meaningless numbers.  I could just have the node server generate an ever increasing random number and it would have just as much meaning.

I also understand the point that you may not have a use for data until you see it.  And if we were talking about a specific sensor reading, I'd agree. Real-time sensor data has meaning.  The example of soil temp has meaning, summing up all the soil temp readings since your station came on line does not.  

I've also been trying to get across the point that making changes/additions to node servers is not "free".  Even what can seem like a simple change can have unintended side effects.  Adding additional code means more testing and more effort over the life of the product.   So if you're going to ask for a change, you should be able to articulate why that change is important. 

[smileyw]

Let me start with this, I agree its not free, it has costs.  Your point is 100% valid.  But accumulated rain is something that most weather systems have (Davis, WF, La Crosse, etc.). And Total rain is the only one on the list that is missing out of this NS.  They do not provide cumulative UV data as an example. So I don't think it is slippery slope you are thinking it is.

Having said that, I have officially given up my crusade on this topic.  It isn't my call to make, and I don't want to end up pissing you off for something that we just simply have different points of view on.  It isn't remotely worth it.  You are the one that has to decide what is worth your time and what isn't, and what will make you money in the end.

I am working on getting the data with a manual variable the has everything before this year and add that to YTD we already have, and I will just manually update it at the end of the year, and it will work.  No big deal.  I already started this last night, but I need to figure out how to get a program to add the NS data to a variable.  I will toy with it when I have some more time.  Maybe tonight, IDK.

[smileyw]

I implemented it with a few variables and a single problem.  I just have to remember to update the "before this year" variable Jan 1st (or around there) to avoid having to back out math.

It is all being sent over to Home Assistant, and I ordered a used fire tablet to show all the data I want to keep track of everything that is going on in the house.

[BamBamF16]

11 hours ago, smileyw said:

I implemented it with a few variables and a single problem.  I just have to remember to update the "before this year" variable Jan 1st (or around there) to avoid having to back out math.

It is all being sent over to Home Assistant, and I ordered a used fire tablet to show all the data I want to keep track of everything that is going on in the house.

Create a program that runs Dec 31 at 2355 and adds year total to variable. 

[smileyw]

Very good call!  Can’t believe I didn’t think of that.🤦‍♂️ That will totally solve that. 

[smileyw]

@bpwwer I have a question.  From a quick look at the code (not knowing Python) it appears that the "yearly rainfall" is year to Date, correct?  (As opposed to a rolling 12 months).  If it is the latter, I need to figure something else out.

 

[bpwwer]

Yes, year to date, and month to date.

[MrBill]

Looks like the discussion has wound down, but "All time Rain" in a tempest is a pretty strange value, and i'm not sure of its purpose.  It's an arbitrary value with no value between stations because one station may have started this year, while the next was "sometime" last year, and another station is 3 years old.

Also my Temptest was replaced by the factory in January, at this point my YTD value isn't even correct.....