Repeat Every xxx -- programs stopped (DST roll over)

[oskrypuch]

I have a number of Repeat Every xxx type logic programs that do a variety of "background" things, for example pull the weather state. On Sunday morning most of these programs had stopped repeating.

I noticed this a day later (today), I have a routine (time of day triggered) that sniffs the temp & baro pressure, and if there is no change over two 12 hour periods, then it assumes an error state and lets me know.

I can't be certain that the DST roll over was the culprit, but the circumstantial evidence (and the logic to it) is suggestive.

I'm running a relatively stable v5.4.4 on a Polisy. IAC, it is an old version, and probably it is no longer supported. I'm just looking for a workaround for future issues, whether DST related or not.

I looked before, but couldn't find a way to do it, is there a way to sense if a program is currently executing? For example, when the Details list would show Activity: Running 'Then', is there a way to pull that state into a program and act on it?

I know you can check a program's last execution result (Then or Else), in fact we all used that many years ago before we had variables, to provide a variable-like True/False capability, but that doesn't do it. I need to know if the Repeat loop is actually executing, rather then how the logic finished.

Setting/resetting a variable in the loop as a semaphore doesn't work, because I'm trying to determine if there was an errant stop to the program.

To get around this issue I could just execute a (re)initiation of all the Repeat Every xxx programs once a day. That would work, but seems programmatically untidy.

Thoughts?

* Orest

 

Edited November 6, 2023 by oskrypuch

[apostolakisl]

10 minutes ago, oskrypuch said:

 

I looked before, but couldn't find a way to do it, is there a way to sense if a program is currently executing? For example, when the Details list would show Activity: Running 'Then', is there a way to pull that state into a program and act on it?

I

 

You could have your repeat set a variable to say 1.  Then you could have another program that sets that variable to 0 on a schedule, like every morning at 8am, or you could have it run more often, but it needs to be a schedule, not a repeat.  Finally, your last program would check periodically to see if that variable is one or zero.  Depending on how often the program repeats and when your program resets the variable you would need to adjust the times for this stuff.  For example:

If
  whatever
Then
  repeat every hour
    whatever you want
    set variable to 1

If
   time is 8am
Then
   set varialbe to 0

If 
   time is 9am
   and variable is 0
Then
   send me an email

 

[oskrypuch]

Yes, that would do it!

Competing programs, when the other wins, you have an error state. A bit of delay, and an indirect check, but it works. Just need to set the timer triggers intelligently.

* Orest

 

Edited November 6, 2023 by oskrypuch

[apostolakisl]

17 minutes ago, oskrypuch said:

Yes, that would do it!

Competing programs, when the other wins, you have an error state. A bit of delay, and an indirect check, but it works. Just need to set the timer triggers intelligently.

* Orest

 

Some of the node servers have a heartbeat, one of the values just goes back and forth between 0 and 1 every 60 seconds.  

If   (run at startup)
  status of node is 1
  or status of node is 0
Then
   wait 5 minutes
      send notification

The value changing every 60 seconds restarts the program so the 5 minutes never finishes unless the node goes offline.

 

You could probably do something like that instead of what I wrote above

If
   whatever you want
Then
   repeat every 1 minutes
      whatever you want
      repeat 59 times
         set variable to 1
         wait 1 minute
         set variable to 0

Now you would have a variable that is a heartbeat every 1 minute and you could just have a second program to send you a notification.  But if you have a concern about it stopping at DST or whatever also add some hard start points for it

If   (run at startup)
  status of node is 1
  or status of node is 0
  or time is 8am
  or time is 8pm
Then
   wait 5 minutes
      send notification

 

[oskrypuch]

And of course, besides sending a notice, you could have it re-initiate the repeating program that has apparently stopped. Need to be a little careful to not set up an infinite loop.

* Orest

[oskrypuch]

OK, implemented, with an error notice and auto-restart, for a bunch of those background threads.

Thanks.

* Orest

Edited November 6, 2023 by oskrypuch

[apostolakisl]

29 minutes ago, oskrypuch said:

OK, implemented, with an error notice and auto-restart, for a bunch of those background threads.

Thanks.

* Orest

It also occurred to me that you might be able to use the action of the program itself as the "heartbeat" depending on what that action is rather than having a variable that serves as the heartbeat.  In other words, if the programming is expected to change the state of something, you can monitor that the state of that thing is changing rather than the state of the variable is changing.

OH YEAH!  The variable you use needs to be a state variable.  I don't think I mentioned that.  An integer variable will not work.

[oskrypuch]

load poly data TP

If
   - No Conditions - (To add one, press 'Schedule' or 'Condition')
 
Then
        Repeat Every  5 minutes 
           [blah, blah, blah]
           $sm_poly_TP  = $xc_true

This is the main program that needs to cycle, that was halted by the DST change (perhaps). It loads up a whole bunch of real time weather data into variables. Added the sm_poly_TP integer variable line.

 

monitor Poly TP

If
        Time is  3:00:00AM
     Or Time is  9:00:00AM
     Or Time is  3:00:00PM
     Or Time is  9:00:00PM
 
Then
        $sm_poly_TP  = $xc_false
        Wait  35 minutes 
        Run Program 'monitor Poly TP.restart' (If)

Every six hours this "service monitor" program resets the same variable, and waits for a Repeat interval + 30min (35 min) to see if it has been set again.

 

monitor Poly TP.restart [disabled]

If
        $sm_poly_TP is $xc_false
 
Then
        Run Program 'load poly data TP' (If)
        Send Notification to 'xxxxxxxxx - email' content '$my_error'

If not, then the restart and email notice is triggered. Although the integer variable can't trigger this program in any case, the program is disabled for good measure.

 

The timing of the set/reset and recheck is dictated by the repeat interval, and how time critical the data is.

Simple, solid. Automated, and logs the issue.

Used the same format for the others.

Thanks again for the idea.

I'm looking forward to the next DST changeover! 😃

* Orest

Edited November 7, 2023 by oskrypuch

[apostolakisl]

I don't really know what all of the names of your stuff is, but if it is loading real time weather data into a variable, you would presume that the data would change at least a little with every update, especially if it is reading to the tenths place.  You could use that as a retrigger for your monitoring program instead of a heartbeat.  As an example, I have wireless tags monitoring a walk in freezer and fridge at my church.  I have a program that retriggers every time the temp changes.  If it fails to retrigger after 45min, I get an alert.  For that program to fail, both the wait clause would have to fail and the wireless tag connection would need to fail essentially simultaneously.  As a precaution, I could add a few "if" items such as if time is 6am, 12noon, 6pm, and 12mn to ensure I don't go more than 6 hours without an alert in the failure event I mentioned.

 

Freezer Online  - [ID 009F][Parent 009D][Run At Startup]

If
        'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is 0.0°F
     Or 'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is not 0.0°F
     Or 'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Humidity is not 0.0%
     Or 'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Humidity is 0.0%
 
Then
        Wait  45 minutes 
        Repeat Every  1 hour 
           Send Notification to 'Lou and Stuart' content 'Freezer Monitor Offline'
           Send Notification to 'Denis Phocas' content 'Freezer Monitor Offline'
 
Else
   - No Actions - (To add one, press 'Action')
 

To test your program you can right click on the program that is supposed to be running in a loop and hit "stop" and then see if your program alerts you.

Edited November 7, 2023 by apostolakisl

[oskrypuch]

Yes, I have a monitor program that checks the loaded temp & baro pressures, and if it is not changing, it alerts me. Likely something is amuck. That is how I was alerted to this instance. But that I have to check over a longer period to be sure, so the competing variable approach is also faster, and a step "higher" in the logic.

Many such background processes do have some sort of logic inconsistency generated, if they fail, so that can be easily monitored and acted on, and I do employ that approach.

But, I also have a number of programs that aren't amenable to something similar to checking the temp & baro, or other resultant logic. So, the competing variable set/reset works perfectly.

* Orest

Edited November 7, 2023 by oskrypuch

[oskrypuch]

        'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is 0.0°F
     Or 'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is not 0.0°F

That is clever logic to check for a change, any change, although it does rely on continued execution itself. As you say, adding time triggers would make it more bullet proof. I'm going to use it, somewhere.

* Orest

Edited November 7, 2023 by oskrypuch

[apostolakisl]

1 hour ago, oskrypuch said:

        'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is 0.0°F
     Or 'Walk Ins / TGOC Walk-Ins / TGOC Freezer' Temperature is not 0.0°F

That is clever logic to check for a change, any change, although it does rely on continued execution itself. As you say, adding time triggers would make it more bullet proof. I'm going to use it, somewhere.

* Orest

Probably overkill doing both is zero or is not zero since the odds of it being exactly at 0.0 and staying at 0.0 are slim to none, but whatever.

[oskrypuch]

Nothing wrong with belt + suspenders.

* Orest