IndyMike Posted July 25, 2010 Posted July 25, 2010 Thought I would share an example of how poor automation programming can lead to physical damage. I came home last week to find that my 1 1/2 inch sprinkler line had ruptured between the house and the sprinkler valve. Fortunately, the system had only been on for 10 minutes. This amount of time created a pond between my house and the walkway. Had this happened the previous week while a was away on vacation, the results would have been catastrophic. Other than a manual shutoff inside the house, there is no way of shutting down the leak. On inspection, I found 2 sections of the 1 1/2 inch line were "light duty" PVC schedule 20 or lighter while the remainder of the system was schedule 40. I replaced the two sections with schedule 40 and everything appeared normal. I then tried to convince myself that 9 years of service combined with yearly "blowouts" with my 2-stage compressor had simply worn that pipe out... Since the water "event" I've reviewed the automation log and have formed a new opinion of the root cause of failure. A number of programming errors had created rapid cycling of the sprinkler valve. This produced water hammer in the line and eventually lead to its failure. System setup I'm using a Hunter sprinkler controller with an IOLinc wired across the rain terminal input. I have sprinkling programs loaded into the Hunter and simply enable/disable the IOLinc at specific times based on Weatherbug data. Program 1 (Evening Sprinkler Control) The following was my starting program. I ran this for approx. 2 months, but never looked at the log file in detail. If From 5:15:00PM To 8:30:00PM (same day) And Module 'Climate' Rain Today is 0 " And Module 'Climate' Temperature <= 90 °F And Module 'Climate' Wind Speed Average < 15 mph Then Set 'Sprinkler-Relay' On Else Set 'Sprinkler-Relay' Off In coding the above, I had ignored my own advice - never use Weatherbug data to directly control a device. I should have used this program as a flag (true/false) and used a separate program to monitor it's status and control the sprinkler relay. Problems with Program 1: 1) The above program triggers whenever Weatherbug data is available. I had my update intervals set at 30 minute intervals so the program would send on or off commands to the IOLinc every 30 minutes, 24 hours a day (powerline pollution). I understood this, but chose to use the program anyway due to lack of time. 2) What I did not realize is that Weatherbug data is parsed as individual entries. In my above program the Temperature and Wind conditions are what I would call "High Rate" variables. These are updated each time Weatherbug data is retrieved. Since they are parsed separately, the program is triggered twice for each weatherbug update. The following is an example of my sprinkler program being triggered twice during a single W weatherbug update: Sun 07/25/2010 11:12:10 AM : CLI-WBug: Connecting to datafeed.weatherbug.com Sun 07/25/2010 11:12:11 AM : CLI-WBug: Successfully Processed WBug Response Sun 07/25/2010 11:12:11 AM : [MOD 2 2 1 1] 739000.000000 Weather - Temperature = 73.9 °F Evening program Trigger (false) on Weatherbug Temperature: Sun 07/25/2010 11:12:11 AM : [iNST-ACK ] 02 62 13.32.8B 0F 13 02 06 LTOFFRR(02) Sun 07/25/2010 11:12:11 AM : [iNST-ACK ] 02 62 13.32.8B 0F 13 02 06 LTOFFRR(02): Process ACK: duplicate ignored Sun 07/25/2010 11:12:11 AM : [MOD 2 2 1 6] 510000.000000 Weather - Humidity = 51 % Sun 07/25/2010 11:12:11 AM : [MOD 2 2 1 10] 550000.000000 Weather - Dew Point = 55 °F Sun 07/25/2010 11:12:11 AM : [MOD 2 2 1 11] 30000.000000 Weather - Wind Speed = 3 mph Sun 07/25/2010 11:12:11 AM : [MOD 2 2 1 12] 70000.000000 Weather - Wind Average Speed = 7 mph Sun 07/25/2010 11:12:12 AM : [MOD 2 2 1 13] 30000.000000 Weather - Wind Direction = NE 2nd program Trigger (false) on Weatherbug Wind Speed: Sun 07/25/2010 11:12:12 AM : [iNST-ACK ] 02 62 13.32.8B 0F 13 02 06 LTOFFRR(02) Sun 07/25/2010 11:12:12 AM : [iNST-ACK ] 02 62 13.32.8B 0F 13 02 06 LTOFFRR(02): Process ACK: duplicate ignored Sun 07/25/2010 11:12:12 AM : [MOD 2 2 1 18] 259000.000000 Weather - Light = 25.9 %/h Sun 07/25/2010 11:12:12 AM : [MOD 2 2 1 19] 62000.000000 Weather - Light Rate = 6.2 %/h Program 2 (Morning Sprinkler Control) July rolled around and I found that I needed a morning cycle of the sprinkler system to keep up with evaporation. I copied the evening program, changed the time constraint, and added prior day rainfall monitoring. The addition of this program is where I caused the damage to my system. If From 3:35:00AM To 6:15:00AM (same day) And Module 'Climate' Rain Today is 0 " And Module 'Climate' Temperature < 88 °F And Module 'Climate' Wind Speed Average < 15 mph And Program 'Rain Yesterday' is False Then Set 'Sprinkler-Relay' On Else Set 'Sprinkler-Relay' Off Problems with the Combination of Programs 1 & 2: 1) The combination of the Morning and Evening programs created a conflict when one program was within its time range and the other was not. If the Morning program evaluated to "true" it would send the ON signal to the IOLinc and enable the system. The Evening program would necessarily evaluate to a false (since it would not meet the time constraint) and would send an off command to the IOLinc. 2) Through sheer dumb luck, the system appeared to work correctly. The order of events were such that the system would enable at the correct time and run. 3) The parsing of the Weatherbug data, and the multiple triggering per event is were I feel the real damage was done. I had increased my Weatherbug update rate to 5 minute intervals which further compounded the problem. The following log shows the overlap of the two programs which caused rapid cycling of my sprinkler valve at 5 minute intervals. With two on/off cycles every 5 minutes and roughly a 1.5 hour run time (90/5*2) I was hammering the system 36 times each time it ran. The pipe failed within two weeks after my adding the Morning program. As I indicated, I was on vacation over one of those weeks. It would seem that dumb luck was again on my side. I've never considered myself a lucky person. Based on these events, I may have to re-evaluate things. Corrected Programs: As i alluded to above, Weatherbug conditional programs should not be used to directly control devices. Instead, the program should be used as a "Flag" or variable and monitored by another program that controls the actual device. The first correction was to change my Monring and Evening programs to "Status" programs: Morning Status: If From 3:35:00AM To 6:15:00AM (same day) And Module 'Climate' Rain Today is 0 " And Module 'Climate' Temperature < 88 °F And Module 'Climate' Wind Speed Average < 15 mph And Program 'Rain Yesterday' is False Then - No Actions - (To add one, press 'Action') Else - No Actions - (To add one, press 'Action') Evening Status: If From 5:15:00PM To 8:30:00PM (same day) And Module 'Climate' Rain Today is 0 " And Module 'Climate' Temperature <= 90 °F And Module 'Climate' Wind Speed Average < 15 mph Then - No Actions - (To add one, press 'Action') Else - No Actions - (To add one, press 'Action') The above programs are then used as "Flags" or conditions for two programs that actually control the Sprinkler valve. This prevents the high rate cycling of the valve. Sprinkler enable: If Program 'Evening Status' is True Or Program 'Morning Status' is True Then Set 'Sprinkler-Relay' On Else - No Actions - (To add one, press 'Action') Sprinkler disable: If Program 'Evening Status' is False And Program 'Morning Status' is False Then Set 'Sprinkler-Relay' Off Else - No Actions - (To add one, press 'Action') Other Corrections: I encountered a condition this weekend where the wind speed was toggling above/below the 15 mph limit of my program condition. As a workaround I slowed the Weatherbug update to 30 minutes. I'm considering separate programs for monitoring windspeed/temperature and Rainfall. The effect that I am going for is 1) If windspeed/temperature are within limits, then run the program and ignore future updates (don't shut down the system while it's running). 2) If it begins raining while the system is running, shut it down. As you might imagine, I'm approaching this cautiously. I consider myself extremely lucky that my mistakes only lead to a split piece of PVC. This amounted to a few dollars worth of plumbing and an hour of "quality time" with a shovel. It could have easily resulted in a flooded basement or a fried pump (~$1200). I am by no means a programmer by nature. I cannot "see" faults in a program through mere inspection. I need to run/observe the operation to determine the functionality. Obviously, I did not perform "due diligence" in my testing of the above. Please do not repeat my mistakes. You may not be as fortunate as I was. After making changes to your system, monitor both your event viewer and log files. If you have questions/concerns, post them here. The ISY team is excellent and there are numerous members who are both "ISY programming savvy" and willing to actually test programs. IM
Brian H Posted July 25, 2010 Posted July 25, 2010 Thank you for sharing your findings. This may assist others to check things thoroughly. Glad you caught it before it got to be a major flood.
Sub-Routine Posted July 26, 2010 Posted July 26, 2010 Hi IM, I think that you would be better to use two separate programs to control your sprinklers, one to start them and one that always turns them off. If the conditions change while the sprinklers are running you could shut them off early and the ensuing program would be redundant. I don't believe the extra PL traffic would impact normal comms, it would ensure that the sprinklers quit, no matter what other conditions. I don't have sprinklers but it annoys me when lights turn on and off every minute. What I do with WB data is to add a Wait so a cloud passing overhead doesn't turn on/off lights. Clear Day If Module 'Climate' Light >= 48 And Program 'Wait to Change Living Room' is False Then Run Program 'Wait to Change Living Room' (Then Path) Else - No Actions - (To add one, press 'Action') When the sky brightens Only runs if sky was dark during last check Cloudy Day If Module 'Climate' Light < 48 And Program 'Wait to Change Living Room' is True Then Run Program 'Wait to Change Living Room' (Else Path) Else - No Actions - (To add one, press 'Action') When the sky turns dark Only runs if sky was light during last check Wait to Change Living Room If - No Conditions - (To add one, press 'Schedule' or 'Condition') Then Wait 5 minutes Set Scene 'LR Stormy Day' Off Else Wait 5 minutes Set Scene 'LR Stormy Day' On Bright Day = True Dark Day = False I hope that gives you some ideas. Rand
IndyMike Posted July 26, 2010 Author Posted July 26, 2010 Rand, That's a slick approach. Effectively using the 5 minute wait to make sure that the condition persists - I like it. I'm assuming that your WB update rate is less than the 5 minute wait period (1 min?). At present, my WB update is set to 30 minutes. Regardless, I am going to use your 5 minute wait as a precaution for those times when I'm playing with the WB update. Thanks, IM
Sub-Routine Posted July 26, 2010 Posted July 26, 2010 Hi Mike, I am using the default 60 seconds. Since the data reflects the current conditions my program provides a buffer. In my example it would have to remain dark for 5 continuous readings before the lights turn on, or vice verse. I think it would work well for your wind speeds and temperature. IMHO It always bothered me to install lighter (cheaper) materials but the contractor and owner always made that decision. Personally, I always thought the labor cost more than offset the material savings. Rand
tullly Posted September 2, 2010 Posted September 2, 2010 How did you program yesterday rain fall program tully
Sub-Routine Posted September 2, 2010 Posted September 2, 2010 ryarber wrote a nice set of programs: Program set for managing irrigation based on rainfall Rand How did you program yesterday rain fall program tully
Goose66 Posted September 2, 2010 Posted September 2, 2010 Another place where the seperation of trigger conditions and status conditions in the ISY makes a ton of sense!
IndyMike Posted September 2, 2010 Author Posted September 2, 2010 Another place where the seperation of trigger conditions and status conditions in the ISY makes a ton of sense! Exactly correct - although in this particular instance it would have made gallons of sense. I had viewed the "rain input" terminals on the Hunter controller as a simple program enable. What I failed to recognize is that, when the the irrigation programs were running, cycling the "rain input" would directly control the sprinkler valves. Rapid cycling of the valve led to repeated water hammer and the failure of the PCV. I have since been using Rand's suggestion of the 5 minute wait, which effectively filters input changes. I've been monitoring for over a month now and have not noticed any cycling (I can see that it should work, but I have to monitor to verify). I don't think I ever properly thanked Rand for his input, and wish to correct that now. Thank you Rand - for a guy who used to work the high steel, you are one heck of a programmer. IM
Sub-Routine Posted September 3, 2010 Posted September 3, 2010 You are welcome IM, I'm glad it works for you! Rand I have since been using Rand's suggestion of the 5 minute wait, which effectively filters input changes. I've been monitoring for over a month now and have not noticed any cycling (I can see that it should work, but I have to monitor to verify). I don't think I ever properly thanked Rand for his input, and wish to correct that now. Thank you Rand - for a guy who used to work the high steel, you are one heck of a programmer. IM
backinthelab Posted July 12, 2011 Posted July 12, 2011 Well, sure am glad I found this before I started to write programs! Still, I'm not quite there yet as I turn into an idiot as soon as I take a new IOlinc out of the box. Would anyone mind telling me how to wire this to the Pro-C controller? IndyMike, you said you wired it across the two sense terminals of the Pro-C? Would that come from the N/O terminal of the IOlinc? How about the COM terminal? Any help would be greatly appreciated, this is one device that still confuses me to the point of frustration! You guys already help a ton on the programming level, thanks!
IndyMike Posted July 13, 2011 Author Posted July 13, 2011 Hello backinthelab, My Hunter controller is actually the Grandfather of the Pro-C (it's a 10 year old SRC controller). The connections on the Pro-C are a bit different, but should still work. Have a look at page 8 of this link: http://www.hunterindustries.com/Resources/PDFs/Owners_Manuals/Domestic/lit329w.pdf. The IOlink will wire in the location for the rain sensor. I connected the NO (normally open) and common terminals of the IOLink to the sense terminals of the Hunter. Polarity doesn't matter - you're using the IOlink as a switch input to the Hunter. When connected to the NO/COM terminals, the Hunter will be enabled when the IOLink is ON (relay closed).
backinthelab Posted July 13, 2011 Posted July 13, 2011 Sweet, thanks! I'll give it a try tomorrow and start programming. As if the IOlinc wasn't confusing enough, it really screwed me up when I saw the two sense terminals on the Pro-c! But, as usual, the brains of the ISY community come to the rescue! Thanks!!!
Recommended Posts