Proposed Evapotranspiration Algorithm in 3.1.4 Beta

[Mark Sanctuary]

It's like a good book and I can't wait to read the ending. Don't leave us hanging to long.

[Michel Kohanim]

Hi Peter,

 

Thanks so very much for detailed explanation. It all makes sense. Once we have your algorightm, I will check against my understanding and then we should have the first cut implemented in the next release. This is looking very good.

 

All,

 

I need your consensus to move this topic to either Green or Questions/Answers (unless you have another suggestion). It does not make sense for us to have this topic in SHN section when they actually have nothing to do with it.

 

Thanks again and with kind regards,

Michel

[TJF1960]

I need your consensus to move this topic to either Green or Questions/Answers (unless you have another suggestion). It does not make sense for us to have this topic in SHN section when they actually have nothing to do with it.

 

Perhaps the "How To" section since there will be explanations on how to implement I would assume.

 

Tim

[Mark Sanctuary]

I think it landed here because of the ezFlora is what everyone thinks of as an irrigation device. It has rather turned into a new feature talk, educational information session, and will end with how to presentation. So I am mixed on where it should go.

[peterd]

My suggestion is re-title the thread "Proposed Evapotranspiration data in 3.1.4 Beta" (as that's where most of the discussion has been) and move to Product Requests. Once implemented we can do a How To thread and if need be have links to the few relevant posts in this thread.

 

Just sitting down to write up more. Stand by...

[Mark Sanctuary]

My suggestion is re-title the thread "Proposed Evapotranspiration data in 3.1.4 Beta" (as that's where most of the discussion has been) and move to Product Requests. Once implemented we can do a How To thread and if need be have links to the few relevant posts in this thread.

Rename to... Proposed Evapotranspiration data in 3.1.4 Beta

Move to... Product Requests

 

Sound good?

[peterd]

Michel -

I've been color-coding some text to get your attention. If you happen to be color-blind please let me know. (This may sound like a joke, but I've had this run into this with color-coding in the past...).

 

First a note to Michel -

Mark & I discussed a small but powerful change to the concept of Daily values, and haven't heard back from you. Here it is in a nutshell:

 

The idea is that much of this revolves around "Daily" averages, tallies, etc. If Daily means once every 24 hours, there should be no reason not to allow that period to be midnight to midnight, noon to noon, 17:27 to 17:27, etc. The value of this becomes clear for users wanting to irrigate at say 11pm. If the Daily period were fixed at a midnight start, they'd have to base irrigation decisions on data that was 23 hour old. A lot (e.g. torrential rain) can happen in 23 hours to affect that decision. OTOH, if they can say "My Daily period is from 23:00 to 23:00" (ok, actually to 22:59:59) then the irrigation decision at 11pm is completely current.

 

The beauty of this is that it simplifies out "today" and "yesterday" values - you just have the "Daily" value representing your previous daily period.

 

Please post ASAP if this will be a problem. I'm continuing this writeup assuming it can be done.

 

 

I've decided to start with a post on variable names. Having had to maintain some code with short non-descriptive variable names, I'm a big believer in making them self-documenting.

 

Of course UDI is free to use whatever internally, but if we have a common "vocabulary" to use I think we'll avoid much of the confusion that has plagued this thread.

 

I'll continue writing up algorithms and post as soon as I can, but meanwhile please ask any questions or make any suggestions now so I can incorporate them right away.

 

Note 1: I've also tried for names which (w/o underscores, of course) can be directly used if the variable is presented in the UI.

Note 2: All variables dealing directly with water are in units of depth (mm internally, entered/presented as mm if Metric presentation chosen in Climate config or as inches if English chosen)

 

 

In the Climate Module configuration:

Water_Applied_per_Irrigation = (Value entered by user)

The amount of water the user's irrigation program applies to the soil during each run.

 

Region_Factor = (Value based on location selection by user)

Adjusts for the differing ET in inland vs coastal areas.

Units: Decimal, range 0.0 to 1.0 Michel - Is this correct ???

 

Soil_Absorption_Factor = (Percentage selected by user)

Adjusts for the relative efficiency of soil absorption (used by ISY only for absorption of rain - the user is responsible for factoring this into the irrigation run-times in their program).

 

Perform_Calculations_By = (Time of day selected by user)

This lets the user decide when the Daily ET average and Irrigation Requirement calculations will be done.

 

Note 3: Actually a presentation suggestion - since this all goes well beyond WeatherBug data, perhaps an "Irrigation" box on the Climate Configuration tab?

 

 

Internal to ISY:

Evapotranspiration_Sample

This is a single instantaneous ET calculation result.

 

Evapotranspiration_Tally

This is the sum of all Evapotranspiration Samples since the last Daily calculations.

 

Sample_Count

How many Evapotranspiration Samples since the last Daily calculations.

 

Total_Rain

Previous_Rain_Total

Allow WB's "Rain Today" (a midnight-to-midnight total) to be used for Daily calculations at the time specified in Perform_Calculations_By.

 

 

Available to users' irrigation programs:

Daily_Evapotranspiration

The average of all Evapotranspiration Samples since the last Daily calculations.

 

Daily_Water_Deficit

The Daily_Evapotranspiration less whatever rain has been absorbed by the soil since the last Daily calculations.

This is a signed float and can be a negative value in the event of lots of rain.

 

Irrigation_Requirement

A running balance of Daily_Water_Deficit since the last time the user's program triggered a decrement or reset.

 

 

OK. I think I got them all. At this point I'd like input on the names, scope and general understanding of the variables. I'd suggest holding usage questions for after the algorithms post. I'm out of time for now, but will post algorithms ASAP.

 

Peter

[Mark Sanctuary]

The post looks good Peter.

 

Perform_Calculations_By = (Time of day selected by user)

This lets the user decide when the Daily ET average and Irrigation Requirement calculations will be done.

I knew where you where heading when you asked if you could add one more variable. I like this idea! The user could set this to 11:00pm to match with a irrigation run at that time. Yeah might cause a loss of the last hour of Rain Today data but I think when its raining hard... the rain numbers would be high and nothing is going to get watered anyways.

[Michel Kohanim]

Hi Peter,

 

I am not color blind (yet)!

 

As far as the Perform_Calculations_By variable, yes, we can add it. This said, I am very much against it:

a. Adds more variables that we have to describe to the user ... what is the procedure/algorithm for users to figure out the optimal value for this variable?

b. It shifts the definition of "today" to a user who may not have the expertise to figure out what this value should be

c. [Daily]_Water_Deficit is a function of Rain Today + Water_Applied_Per_Irrigation ... Rain Today is reset at 00:00 by WB so the worst case scenario would be +/- 3 hours in the U.S. (based on the station). With the other approach, the offset could be +/- 23:59 depending on what the user decides

 

My proposal is to use WB's timezone for that station, do calculations at 00:00 (station time which, in 99.9% of the cases, is equal to ISY timezone) and let today remain "today" and not "3/4 today and 1/4 yesterday", etc.

 

I still need to see the overall algorithm and make sure I understand it.

 

With kind regards,

Michel

[Mark Sanctuary]

Thinking about Perform_Calculations_By variable... I actually now think (after Michel has pointed out some good points) syncing it with the WeatherBug clock actually sounds like the best option.

[peterd]

I have one version of the algorithms written up, but will wait to post them until:

1. We get resolution on the notion of user-defined calculation time

2. I get some feedback on the variables (names, range, scope & high-level description).

 

The algorithms may need to be changed depending on the feedback for these 2 points.

[...] I actually now think [...] syncing it with the WeatherBug clock actually sounds like the best option.

OK, let's see - where's the smiley for "Throws hands up in surrender?"

 

Michel -

I started out arguing for all calculations to happen at 00:00 (actually 23:59:59), so I'm fine if we go that way.

 

Mark pointed out that if you want to begin irrigation late in the day (say 22:00 or 23:00) your decision could be very much out of date. Here's an example: At midnight, Irrigation Requirement reached your threshold - so your program will water the plants - but it was pouring rain all day. So, if you did the calculations now (some 22 or 23 hours later) they would have your irrigation wait for another 2 or 3 days!.

 

I realize that the user's program could try to second guess the Irrigation Requirement calculations (e.g. by looking at whether Rain Today were over some value - which would just be a best-guess) but I have a feeling it's going to get very convoluted. On the other hand, I don't think the alternative has all the drawbacks you describe:

a. Adds more variables that we have to describe to the user ...

I agree that this one variable ("What time of day do you want your irrigation calculations performed?") needs to be described (oh wait, I just described it ). But I would argue that things get simpler because we completely eliminate the notion of "Today" and "Yesterday" in favor of "Daily" (meaning "For the last complete 24 hour period").

what is the procedure/algorithm for users to figure out the optimal value for this variable?

Good news - there is no optimal time! Most users would probably leave this at midnight, but changing it would be purely personal preference. The beauty of the Irrigation Requirement Balance method of irrigation scheduling is that all it cares about is that the balance is calculated/updated (once) every 24 hours.

b. It shifts the definition of "today" to a user who may not have the expertise to figure out what this value should be

I continue to argue strongly that no matter what time the calculation is done we avoid all notions of "Today" and "Yesterday" within the irrigation subsystem. Note that in the variables I've proposed, there are no "Today" or "Yesterday" variables, only Daily_. All the user needs are Daily values. These are by definition updated only once in every 24 hour period, so any Daily value is the value as of the end of the last complete 24 hour period. (Assuming calculations @ midnight, the last complete day.) There is no "Today" value of any use because the Irrigation Requirement Balance can't be updated until the rain for the past 24 hours is known.

c. [Daily]_Water_Deficit is a function of Rain Today + Water_Applied_Per_Irrigation ...

Actually, Daily_Water_Deficit is a function of Rain Today and Daily_Evapotranspiration - but I know the point you're making is the effect of Rain Today.

Rain Today is reset at 00:00 by WB so the worst case scenario would be +/- 3 hours in the U.S. (based on the station). With the other approach, the offset could be +/- 23:59 depending on what the user decides

IFF we implement the Perform_Calculations_By setup, I already have the 2 lines of code which would allow the internal calculations to take this into account and tally the rainfall over the previous 24 hours no matter what time the user decided their daily calculations should occur.

 

With this in mind, please think about it for a bit while I'm writing up the algorithms. If you still feel strongly that it shouldn't be done, say so & I'll update the list of variables and the algorithms back to 00:00 only.

 

Any input as to the variables I've proposed? If we did implement the user selectable calculation time, I now think it would be better if we called it Perform_Calculations_At.

 

Peter

[Mark Sanctuary]

OK, let's see - where's the smiley for "Throws hands up in surrender?"

[peterd]

Double Facepalm

 

Mark -

 

Thank you for the huge grin. Still hasn't gone away...

 

Peter

[Michel Kohanim]

Hi Peter,

 

Thank you but I am still not convinced mostly because I do not understand this statement (mixture of past, present, and future in present tense):

 

At midnight, Irrigation Requirement reached your threshold - so your program will water the plants - but it was pouring rain all day. So, if you did the calculations now (some 22 or 23 hours later) they would have your irrigation wait for another 2 or 3 days!.

 

If I understand what you mean, you are just basically shifting the time and the same exact problem (as above) CAN happen at any time depending on the relationship between watering plants and daily calculations. In short, for your example above to work consistenly, one has to be omnipotent to know exactly when it is going to rain and how much in relationship to daily calculation time and watering time (i.e. some 22 or 23 hours later ...).

 

There's one more comment you made that makes me a little worried:

At midnight, Irrigation Requirement reached your threshold - so your program will water the plants ...

 

I am hoping that you are using a scheduled (say from 8:00 PM to 11:59 PM) program to check the threshold for watering the plants. Correct?

 

I do apologize for having such strong feelings about this: at the end of the day, this feature has to be supported, we have to have our beloved tech support team trained, and we have to be able to provide easy to understand answers. At the moment, I really do not see an easy way of describing this value.

 

With kind regards,

Michel

[peterd]

Hi Peter,

 

Thank you but I am still not convinced mostly because I do not understand this statement (mixture of past, present, and future in present tense):

 

At midnight, Irrigation Requirement reached your threshold - so your program will water the plants - but it was pouring rain all day. So, if you did the calculations now (some 22 or 23 hours later) they would have your irrigation wait for another 2 or 3 days!.

Michel -

 

Let me try to illustrate using a timeline. In this example, you've decided to irrigate whenever the amount of water loss reaches 25mm.

 

Scenario A - calculations always happen at 00:00, your irrigation program runs at 23:00:

1. 00:00 - Irrigation Requirement is updated. Its value is now 25mm.

2. Between 00:01 and 22:59 it rains. Total effective rainfall = 20mm. Maybe it falls all at once, maybe on-and-off, maybe non-stop all day long. It doesn't matter for the example (what matters is that the rain occurred between updating Irrigation Requirement and testing its value).

3. 23:00 - The IF schedule condition in your irrigation program evaluates true. The ANDed test of "Irrigation Requirement >= 25mm" also evaluates as true. Your program runs through the zones which you've timed to dispense 25mm of water. In reality, because 20mm of rain was absorbed between the time the calculation was done and the IF in your program evaluating Irrigation Requirement you've overwatered by 20mm.

 

Scenario B - you've scheduled your irrigation program to run at 23:00, so you set the calculations to happen at 22:59:

1. Between 00:01 and 22:59 it rains. Total effective rainfall = 20mm.

2. 22:59 - Irrigation Requirement is updated. Its value is 20mm less than what it would have been at 00:00 (before the rain).

3. 23:00 - The IF schedule condition in your irrigation program evaluates true. The ANDed test of "Irrigation Requirement >= 25mm" evaluates as false. Your program completes without watering anything.

 

If I understand what you mean, you are just basically shifting the time and the same exact problem (as above) CAN happen at any time depending on the relationship between watering plants and daily calculations. In short, for your example above to work consistenly, one has to be omnipotent to know exactly when it is going to rain and how much in relationship to daily calculation time and watering time (i.e. some 22 or 23 hours later ...).

You've identified the problem (a significant delay between updating and evaluating Irrigation Requirement). As I hope the new example shows, the whole point of a configurable calculation is to close that gap for anyone who doesn't schedule their irrigation to start just after midnight.

 

I do apologize for having such strong feelings about this: at the end of the day, this feature has to be supported, we have to have our beloved tech support team trained, and we have to be able to provide easy to understand answers. At the moment, I really do not see an easy way of describing this value.

 

No apologies, that's certainly a valid concern. If we believe that there is value to being able to configure the calculation time (because it frees users up to schedule at any time, not just in the very early AM), then let's go from there.

 

I do have an idea for an alternate implementation which I think may be much easier to understand:

 

- No config variable for calculation time

- Instead, a SYSTEM command to trigger calculation

- At whatever time the user wants, their irrigation program asks ISY to perform the calculations

- It then evaluates (the freshly calculated) Irrigation Requirement to determine whether to water yet.

 

I think users already understand having a schedule condition in their irrigation programs. Since we're adding all these calculations to the ISY, asking them to add the trigger of calculations into their programs seems like a straightforward requirement which needs little explanation beyond "Here's how you ask ISY to perform the calculations...".

 

What do you think of that idea?

 

Peter

[Mark Sanctuary]

After I have slept on this another night... I think storing the Daily tally each day into a Yesterday variable is much less confusing, all the calcs can just stay synced with the Rain Today data. If you start the irrigation run at 01:00 and use the Yesterday data your only 1 hour old after the calcs. And if you start the at 23:00 your only 23 hours after the calcs. Either way 1 hour or 23 hours later the irrigation run will be correctly launched 99% of the time.

[peterd]

Mark -

 

I know I haven't posted the algorithms yet, but as you can see from my post about the variables, there aren't any "Yesterday" & "Today" variables. Even if the calculations are fixed to 00:00, then that's when the Daily_ values are set. Here's your statement re-written accordingly:

 

I think storing the Daily tally at midnight each day is much less confusing, all the calcs can just stay synced with the Rain Today data. If you start the irrigation run at 01:00 and use the Daily data your only 1 hour old after the calcs.

 

Having said that, what do you think of the idea that your irrigation program could request the calculations when it runs?

 

Peter

[Mark Sanctuary]

Peter (putting words in Mark's mouth) wrote:

Have you seen Bedtime Stories with Adam Sandler? In the end of the movie when he gets

for him as he speaks because he is the only one that can understand him. This situation reminds me of that.

 

Having said that, what do you think of the idea that your irrigation program could request the calculations when it runs?

If we have that trigger then we will need to make a copy of Rain_Today so we can use it any time of the day along with the rest of the calculations. I guess triggering the calculations from the code would be OK. But what will be displayed in the WeatherBug GUI module then... all the vars the last time they where calculated?

 

 

About the naming, Daily is confusing. The WeatherBug data is using the Today concept so to stay in sync with it we should in the ISY as well. Here in this screen-capture is how I think about the time-frame vars...

[Mark Sanctuary]

Also what do we do if the program does not trigger it every day at the same time?

[peterd]

Mark -

 

LOL re Adam Sandler

 

1. Water_Deficit can't be calculated until a rain total for the previous 24hrs is available, so in your example, it would exist only in the "Yesterday" column.

 

2. I question the assumption that there'll be any "Today" values to see (in my ISY, if Evapotranspiration_Average were calculated every time the WB vars were fetched, that would mean 2779 needless calculations (just in case I wanted to see a number changing in the UI) for every 1 useful calculation). When you grayed out that variable, were you indicating that it's just for display and there's no other need for it as a moving average?

 

3. If the only "Today" variable that's visible is the WB Rain_Today, then rather than have all the irrigation variables (which I would argue should be in the climate module, but visually set off from the WB data in the UI - since they are ISY values, not WB data) be "Yesterday", isn't it just as valid that they be "Daily"? (Which I think would be a lot less confusing if the calculation time was up to the user's program...)

 

4. The user will be instructed to write their program (I actually envision writing up and delivering a sample program) so that it is scheduled at the same time every day. The example was given, though, of sometimes needing to delay starting for an hour or two because of a late party. Say they did happen to run it an hour or two late, then on one side their calculations would be somewhat short (22 or 23 hours) and on the other a bit long (25 or 26 hours). That would all even out by the next cycle. If they're completely ignoring the intent and running the irrigation program at random times , that's their problem.

 

I can see that I have to go ahead and post the algorithms I have so far which work on the assumption that the calculations could be triggered by the user's program. Unfortunately, I can't get to that for a while. Bear with me please...

 

Peter

[Mark Sanctuary]

1. Water_Deficit can't be calculated until a rain total for the previous 24hrs is available, so in your example, it would exist only in the "Yesterday" column.

Yeah I think it is Yesterday and there is no today data. I will fix the screen-capture.

 

2. I question the assumption that there'll be any "Today" values to see (in my ISY, if Evapotranspiration_Average were calculated every time the WB vars were fetched, that would mean 2779 needless calculations (just in case I wanted to see a number changing in the UI) for every 1 useful calculation). When you grayed out that variable, were you indicating that it's just for display and there's no other need for it as a moving average?

Today values are just what is collected for the day and averaged. Yes its visual and it is there for when the copy to yesterday happens, this is what the grey means.

 

3. If the only "Today" variable that's visible is the WB Rain_Today, then rather than have all the irrigation variables (which I would argue should be in the climate module, but visually set off from the WB data in the UI - since they are ISY values, not WB data) be "Yesterday", isn't it just as valid that they be "Daily"? (Which I think would be a lot less confusing if the calculation time was up to the user's program...)

But it is easier to understand that I am basing my irrigation needs off of yesterday's data. Daily sounds to open, is it daily from last week, or next week, what day is it from? Yesterday there is no question about what day the data is from.

 

4. The user will be instructed to write their program (I actually envision writing up and delivering a sample program) so that it is scheduled at the same time every day. The example was given, though, of sometimes needing to delay starting for an hour or two because of a late party. Say they did happen to run it an hour or two late, then on one side their calculations would be somewhat short (22 or 23 hours) and on the other a bit long (25 or 26 hours). That would all even out by the next cycle. If they're completely ignoring the intent and running the irrigation program at random times , that's their problem.

With the difficulty that already comes from understanding how "everything is an event" programming with ISY programs I don't think we need to add more complexities to it. We want to take as much of the guess work out of the hands of the end user. Yes predesigned programs will be helpful but in the end how do you keep the user from breaking the correct functionality.

[peterd]

Yeah but Water_Deficit is not a STORED history, when calculated, it is today's data and should be named that way.

Sure, but if it is being calculated at 23:59:59, it is only today's data for 1 second. Oh, I see you got that (in my best Emily Litella voice) "Never mind..."

Today values are just what is collected for the day and averaged. Yes its visual and it is there for when the copy to yesterday happens, this is what the grey means.

OK, but so far none of the algorithms I've written up (either in the last round or what I will post next) have minute-by-minute running averages or totals.

 

But it is easier to understand that I am basing my irrigation needs off of yesterday's data. Daily sounds to open, is it daily from last week, or next week, what day is it from? Yesterday there is no question about what day the data is from.

Again, this concept only works if the 24 hour period being used is midnight to midnight.

 

With the difficulty that already comes from understanding how "everything is an event" programming with ISY programs I don't think we need to add more complexities to it. We want to take as much of the guess work out of the hands of the end user. Yes predesigned programs will be helpful but in the end how do you keep the user from breaking the correct functionality.

Completely agree about the current complexity. I don't agree that having a line in your program which says something like "Module 'Climate' Perform Irrigation Calculations" (using whatever syntax Michel implements - this is not meant to represent actual syntax) adds the same level of complexity.

[peterd]

Mark -

 

I want to offer the following "food for thought" on "nice to see" variables updated throughout the day:

 

The ISY has X amount of hardware resources (CPU cycles, RAM, etc.). While each additional calculation may use a minute amount of processing power & memory, this is by far not the only functionality yet to come. In fact, we constantly give kudos to UDI for continuing to enhance the ISY software. I'd hate to see a time come when that last feature didn't fit by a couple of bytes or a couple of processor cycles because we had chosen to have an extra few variables, or because of a value updated every minute (or more) instead of once every 24 hours...

 

Peter

[Mark Sanctuary]

You mean this video ...

... love it! ha ha ha!

 

Version two with including existing names and some proposed compromises...

 

 

X amount of hardware resources

The "nice to see" var your already calcing each sampling (ET_today = ET_samples / samples_count). This could be performed only when the GUI tab is viewable. It's a nice to have, if it does not make it in, I won't die.

[Michel Kohanim]

Hi Guys,

 

Apologies for tardy reply ... didn't really know how to reply.

 

I have been thinking about this whole thing. Irrigation Requirement is accumulative and so is Water Deficit. So, at each moment in time, these values reflect how much water has been applied and how much is needed.

 

Now, here's the kicker: why do we even need to calculate daily or yesterday's? The only difficulty would be to figure out what we mean by Rain "Today".

 

Peter, we can add that command (just like reset and cycle complete) but, as Mark said, I think it's just going to cause confusion.

 

And, yes, I think I need to see your new algorithm before submitting my thoughts. I am still unclear as to how we would calculate the total amount of rain per 24 hour period if our time is not the same as the station's time. Well, unclear is not a good word. Basically, we would have to have keep subtracting the amount of rain from the previous 24 hours from the amount of rain for today thus far.

 

With kind regards,

Michel