Creating a Power Management Console for the ISY ZS option with Node-Red

[LFMc]

Home Energy Management Dashboard Project

 

This project was to create a “Dashboard” that I can see and manage the energy consumption in my home. As the saying goes, “if you can’t measure it, you can’t improve it” –Peter Drucker.

 

My objectives were to find a easy way to build the dashboard that could be used on tablets, devices or PCs to monitor real time and historical energy consumption numbers. Also to have some interaction with the system that controls the energy usage (i.e. my ISY ZS) so that I can make some manual overrides. I am not a hard core programmer, so I wanted to find a easy way to make and manage this program. I also wanted to involve as little new or dedicated hardware as possible to keep this inexpensive and duplicate if possible.

 

I really did not want to involve the “cloud” as I do not want an internet outage to throw off my real time management. In my case a well timed internet outage could cost me money if my management system was not able to respond in time.

 

I looked at using Google Docs and the Google Spreadsheet, but that was going to be more complicated than I wanted. I was digging around in the UDI forum and ran across a product called Node-Red and then I ran across Sandy Siebert’s excellent article of setting up a very basic Node-Red server. While it can run on multiple platforms, I decided to try it on my son’s Raspberry Pi (original version) and that way I could learn a bit more about the RPi OS and features. Besides, I have some other things I might like to migrate to the Pi later on. Once I had it working, I bought my own new Pi for about $35.

 

Here is a link to Sandy’s topic on Node-Red. https://forum.universal-devices.com/topic/22799-note-red-chart-and-gauge-tutorial/ .  You can learn some of the basics of getting Node-Red up and running along with some simple node processes to start with. Node-Red is an object oriented programming language that makes it easy for once a year coders like me to get something done. But it has some powerful functions that can include custom Java scripts if needed. There is a vast library of custom nodes and custom flows that really help the beginner.

My goal of this topic is to show you what can be done and give you a few tips in getting it done. It is not to teach you how to run a RPi or how to program in detail Node-Red. It is going to take some trial and error on your part for that. I hope that it does encoruage you to get something like this working for your HA system.

 

I invite comments and feedback as well as questions. I repeat, I am beginner on Pi and Node-Red so I am limited on how much I can help you.

 

Recommended starting equipment/software:

1. Raspberry Pi 3 with SD memory and connectivity to your PC and internet. I use the Raspian OS, version Jessica with the latest patches and updates. It includes Node-Red on it, but be sure to update Node-Red to the latest.

Node-Red runs in a “client-server” configuration to use an old tech term. The Pi becomes the server. You program Node-Red flows and view the dashboard from any browser on your network.

2. In my case, I have a ISY-994i with a ZS option. The ZS option gives it the ability to continously quiery the smartmeter and give continual overall energy consumption feedback. Without this, you would have to use a 3^rd party system. I am only interested in overall consumption, not individual power legs in my circuit breaker panel. Be sure to check to make sure your TDU (Transportation Distribution Utility) supports or allows monitoring of the energy flow via Zigbee SEP. UDI has a list of TDUs they have gotten approval from or at least know they work with them.

 

My needs for energy management are going to be very different than your needs, so your dashboard and Node-Red flows will differ greatly from mine. Mine is totally set up to take advantage of contractual usage minimums that require hitting a very specific usage goal every month. So my typical electric bill runs less than $30 per month 9 out of 12 months in the year due to me accurately hitting my contract goal. I have a 3500’ home in N. Dallas where we do have some extremes in temperatures in the winter and summer. So to be specific, the goal of my energy management system is to hit 1000 KWhr per month and not go under that, but also not to exceed it by much if possible.

Based on those needs, here is a screen shot of my finished dashboard:

 

 

The 4 major areas of the dashboard are:

1. Real Time Monitoring:  This area shows the continual, real time, usage of energy in various gauges that change colors from green to yellow to red. 

2. Historical Information: Charts showing historical usage broken into 3 areas, 30 second instant usage peaks, hourly usage averages and daily usage averages.

3. Buttons:  This gives the ability of the observer to interact with the charts and ISY. It also makes it easy for my wife to have a button that resets our water fountian timer every 7 days when we refill the fountian. (-:   So there are 5 buttons that control the Node-Red charts and ISY functions. The buttons are purposely put down low on the dashboard so that they are typically off the bottom of the screen to keep away prying eyes and fingers from the grandchildren. So you have to slide the dashboard up in most browsers to even see the buttons.

4. Power Shedding: This area gives me feedback on the monthly power shedding feature I have programmed into the ISY.  At a predetermined time during the montly power cycle, the power shedding feature is turned on by the ISY. Once that happens then the two 1 Kw loads will come on for one hour cycles to bump up usage as needed to meet the daily and monthly power goals. The loads are two strategically placed electric space heaters who add heat to the house to off set the use of the main natural gas forced air heating system. At the bottom are various statistics that relate back to power shedding.

 

The actual Node-Red flow chart is attached below in the graphic object oriented format. It shows the logic and programming required to get the dashboard to work. I tried breaking it out into logical sections, but items like the chart history back up and restore required a bit more intertwining of the spaghetti lines to not duplicate a lot of work. Node-Red does a great job of keeping it’s own charting history on the Pi, but only if you don’t restart the Pi, ever. So to off set this, if you need to keep your history for a while after a restart, you need to output the chart history to a text file on the Pi or to a database on the Pi. Mine are simple, so I use 3 text files that get restored when the Node-Red server restarts.

 

There are a few of the function nodes, especially on the chart history save and restore, that contain some java script that are beyond my programming understanding. I found them in Node-Red flow examples in their library at https://flows.nodered.org/ and simply copied them into my flow.

 

Node-Red programming can be exported and re-imported easily so it is very transportable. You can import nodes or entire flows with cut and past. My flows will not work with your set up, but could be used to help you better understand how the data is pulled in and manipulated to produce the desired output. I will plan to put some of the flows into the forum topic as additional interest is expressed.

I hope this gets you going and flowing with Node-Red programming.

 

Displaying the Node-Red Dashboard:  I am currently using an old iPad 2 that stays plugged into power all the time and placed it in the kitchen/breakfast area for all to see. It has generated great discussion and for the first time in a couple of years it has engaged my wife in better understanding the goals and actions taken to manage the power. She is better informed and much more involved in helping to meet our goals. 

              

[fisix]

This is terrific.  Once I can get my hands on a SEP enabled ISY or upgrade, I'll be all over this.

[sandryseibert]

Node Red Power management Dashboard Flow.jpg

 

That's what I link on cooperate. With your ideas I was able to create my own file store per month and now I can see monthly or early data.. cool!

[fisix]

Looks like I'm ready to get started.  ZS linked and retrieving data. 

[fisix]

Alright Leon, 

Getting started, where's the best place to start to figure out how to poll the 994i and extract the data it's getting from the meter, similar to just your 30s trigger section?  

EDIT--> I see your link to Sandry - I'll look there for this.

Next, same question as to how to push control states back to the 994i - do we just have the ability to change custom/abstracted state variables that we've set up on the 994i, or do we have direct access to the state of the devices themselves?  I see benefit to using abstracted state variables regardless - timing is (mostly) handed off to the 994i, and most changes to functions can happen on the 994i without having to mess with the Node Red stuff.

Clearly, I'm starting from scratch.  My plan is to write up the process so that someone interested in smart meter based monitoring and slightly more complex usage-based control of loads (using a 994i) can get a decent minimal system set up with an RPi and little effort - that way they can use what's there as a guide for further efforts and can focus on the harder stuff and making it look nice/wife factor.

Thanks!

[fisix]

Alright, read up on Sandry's guide, and perused the rest interface.  

Because I have a sonology box, i'm going to try setting that up first - basically it'll be a dumb graph, a gauge, and two control buttons - one to poll fast or normal, and one to control a state variable.  Once that's up and running, I'll move to RPi so that I can perhaps have it running side by side with Owncloud.  Excited!

[fisix]

Oof.  There has to be a better way to convert the http get output to numbers that both the graph and the gauge like.  And easier ways to bound things.  It's a bit more difficult it seems if you have negative numbers (for when the solar system is generating).  

Anyway, I have a chart and a gauge showing instantaneous demand.  Looks very yucky so far, and clearly I need to go through a node red layout tutorial.

In any event, I'll add a button to switch between slow poll (30s) and fast poll (2s), and then I'll start making it pretty, storing data, and maybe adding controls to adjust graph view.

[fisix]

One thing.  You have multiple Read Emeter instances triggered by the same trigger point.  I think you may want to just use one, get the whole shebang, and then feed that one to multiple parsers  (the var val stuff).  For example, I have one GET and I can parse the output of that for different values.  

Is there a reason to have them separate?  The timing is a little undefined the way it's done now, and it may be banging your 994i more often than it needs to (unclear if that matters).

[LFMc]

On 6/9/2018 at 3:14 AM, fisix said:

One thing.  You have multiple Read Emeter instances triggered by the same trigger point.  I think you may want to just use one, get the whole shebang, and then feed that one to multiple parsers  (the var val stuff).  For example, I have one GET and I can parse the output of that for different values.  

Is there a reason to have them separate?  The timing is a little undefined the way it's done now, and it may be banging your 994i more often than it needs to (unclear if that matters).

Thanks for the feedback. I have different HTTP get requests on the 30 sec trigger because each one calls a different variable. Except when I call for electric meter data via the rest command then I parse it for just what I need.  I have no idea how you get a bunch of different variables from an ISY with one get or that it matters. 

[fisix]

I've just started having trouble with the meter readings after a few days of bullet proof performance.  I think I moved the ZS too far away from the meter, but I guess we'll see.

Below is an example of a single get used to get the full array of data.  I have two only because i'm trying to only ping data that updates ~once daily every hour or so.  I'll update as things get better.  I think the guide and RPi image I'll put out will use something similar to the below so that someone can just plot all the available data, see how its done, and then modify as desired.  Clearly, I still need to implement your saving data and retrieval, and there are some other glitches i need to fix too. 

Note - the get command i use to retrieve all data is ISY:rest/emeter/metering

 

[LFMc]

2 hours ago, fisix said:

I've just started having trouble with the meter readings after a few days of bullet proof performance.  I think I moved the ZS too far away from the meter, but I guess we'll see.

Yes, I had problems with the ISY Zigbee range also. I could barely get 30-40 feet from my meter (across my garage) and my data would drop out and get errors reading the meter. After a lot of testing, I had to get a SEP Zigbee repeater (from Smartenit.com)  to extend my distance for my ISY. There were still problems, so I put a higher gain antenna on my ISY Zigbee interface and even then I had to modify the antenna for more gain by adding a semi-parabolic non-ferrous, highly reflective, planar array that I had picked up at the local grocery store. (a.k.a aluminum pie pan). That worked great and the pie that came with it wasn't bad either. ?

Being you get most of your input from the meter, you can do what you did with the single query per trigger. Your flow is very clean and simple, good job.  

[paulbates]

I've had the same issue with non-ISY AMI zigbee products. DTE provides us a device that can talk to the meter, update a "cloud" service and provide running & real time app usage via an app. The only place I could get it to work reliably was in a TV cabinet that is directly behind the outside wall that the meter is mounted it.

My ISY ZS was further away and could see the meter, but ultimately DTE did not allow me to connect to the meter so I've never been able to use it

Paul

[Michel Kohanim]

All,

We have over 100 units in a pilot for California Energy Commission/Southern California Edison and are constantly monitoring meter connectivity. Out of 100, only 52 have had bullet proof performance/connection. Out of the rest, the majority are related to the meter changing channels or, simply being out of range. Please note that the Zigbee radio and antenna on ISY have been tested and work for at least 200 meters. Based on these findings, in In 5.0.13 firmware, we use a longer keep alive and then try a rejoin. In more than 80% of the cases, the rejoin succeeds. i.e. the meter changed channel but ISY didn't hear it (or the meter didn't send it), ISY rejoins (scan/search for a new channel), and everything works. The other 20% is extremely random and have not yet been able to figure out the root cause (even with range extenders). This said, out of the 20%, the majority come back on their own after 24 to 48 hours and thus making me speculate that the issue is still the channel change and the new channel having major interference with other things at home and, especially, WiFi routers.

So, LMFc's comments with regards to the antenna make me speculate that the channel change issue is indeed the root cause. i.e. a pie pan has a much wider range for support of different channels than a dipole one. 

With kind regards,
Michel

[LFMc]

56 minutes ago, Michel Kohanim said:

So, LMFc's comments with regards to the antenna make me speculate that the channel change issue is indeed the root cause. i.e. a pie pan has a much wider range for support of different channels than a dipole one. 

Thanks for the detailed feedback Michel. This is good info. As I rely on the meter data continuously throughout the day, having it disappear even for a few hours is problematic for me. And you are correct, my latter problems seem to be intermittent and would typically correct itself after a few hours or a day and then catch back up. Most people may not even notice the outage, but I was watching the REST logs and could clearly see it and it would send bad data to my power management programs (due to my data parsing and lack of error checking in my code). 

But, that being said, when I was initially setting up my unit (as we have discuss previously), I really was having connection issues well below 50', as in no connection what so ever. I still think it is caused by the density of Zigbee and Wi-fi devices so close to my home. I can see a dozen wi-fi routers and I am guessing at least 8-10 Zigbee SEP devices well under 100 yards.  This article refers to this issue https://support.metageek.com/hc/en-us/articles/203845040-ZigBee-and-WiFi-Coexistence. I know both Wifi and Zigbee channel hop and  I know that Wi-fi also lowers transmit power when it is in an area of multiple routers to avoid issues. I was guessing the Zigbee smart-meters might do the same. So testing a 1:1 connection without all the interference might produce much better results than the situation I am in. 

BTW, I started out using a Teflon coated frying pan first, but my wife wanted it back.  

 

[fisix]

Good info.

When your ZS wasn't able to get data from the meter, did your utility de-register your device or did you have to restart the registration process?  When I view my device on the SCE website, it's listed as "registration failed"... so i have an email into them to see if once the communication starts, if I'm supposed to go back into their web page and complete some extra step.

If they're going to require a phone call or more every time the signal drops out (regardless the reason), that's going to be a problem.

Here's the distance stuff:

Original good connection was while the ZS was on the other side of the exterior wall from the meter - panel back, stucco, wood, probably no insulation.

Ran fine for a few days when I moved it back into a closet - add an additional 25feet, 2 exterior walls, and a closet door.

I've placed it back near the original good spot, and I'm stuck in a "Service Discover" loop. - it flashes "rejoining" and then "established" every now and again, but doesn't synch.

Thanks for any thoughts or help.

 

[fahrer16]

I've been using node-red for something similar for a little while now.  I started off using the node-red dashboard but I wound up migrating to an influxDB with grafana as a front-end because it's much more powerful and looks prettier.  I still use node-red to get the data into the database though.

I wrote an ISY node for node-red that gets the data from the ISY for any ISY node, program, or variable.  I also have a small node-red dashboard built that uses that interface and also includes some buttons on the dashboard that send commands to the ISY.  I'd be happy to provide some more detail if anybody was interested in going that route.

 

Data into InfluxDB:

Example Grafana dashboard:

Example Node-Red Dashboard config:

Example Node-Red Dashboard with controls:

[LFMc]

2 hours ago, fisix said:

When your ZS wasn't able to get data from the meter, did your utility de-register your device or did you have to restart the registration process?  When I view my device on the SCE website, it's listed as "registration failed"... so i have an email into them to see if once the communication starts, if I'm supposed to go back into their web page and complete some extra step.

I would guess that every TDU is different from the other in how they handle registration. Like you, I put my ZS right next to the meter until it registered the first time. I have never had to register again, but it is pretty simple for me with Oncor as my TDU.  I can't imagine what it takes to get de-registered.  But it does seem that you might need to start the registration all over again.  

Once connected, I would move it back to the next room, one room at a time while watching the data stream with a rest command on my PC to see if it was getting a good data connection. http://<your private IP address>/rest/emeter/metering/log. That was how I knew I finally needed a repeater and a better  antenna.

[fisix]

Thanks Leon.  I had the ZS in the closet for a good... 4 days? without a single data hitch (as reported by node red), and then it just died.  I'm guessing there may be an issue at SCE, and I have an email out to them to troubleshoot, but I'll poll the log as you suggest to see what gives.  if the ZS is having trouble keeping up with channel changes, i may twiddle with the ZS module settings as Michel noted.

And, thanks fahrer16 - I'm interested, as that looks very nice.  I found a general ISY node-red collection of nodes (just searched for ISY) - is that yours?  Also, am I correct that you're using Bruletech sensors to get current readings?  Also, is influxDB with grafana fairly easy to get going on an RPi?  The end result here, if I can get this all working better, is an RPi image file that can be downloaded by a new user with a ZS and booted by a user with minimal tweaking to get a view of their meter-derived usage data.  I'll still want the grafana info for my home project, regardless, but i'm trying to corral everything into something that could be run on a RPi (hopefully also able to run Owncloud and certs provided and auto updated by LetsEncrypt, but that's probably beyond the scope of the project).  

-fis

[fahrer16]

@fisix, the node-red module I was referring to is called "node-red-contrib-isy".  I don't think there are any others so it's probably the one you're seeing.

I'm using a Brultech ECM1240 but instead of ZS I have it connected to ethernet to Node-Red, which parses the data and sets variables on the ISY.

I have Node-Red running on a RPi but I was worried about corrupting the RPi's SD card by continuously writing to it so I'm hosting my InfluxDB and Grafana instances on a Debian machine.  I'm probably being paranoid about the SD card for nothing, after some searching it seems a lot of people are hosting an InfluxDB on their RPi.  I found a pretty good install guide: https://www.circuits.dk/install-grafana-influxdb-raspberry/

The function I'm using in node-red to format the data from the ISY device and variable nodes before feeding into the InfluxDB batch node is:

var input = msg;
var output = {};

output.payload = [{
    measurement: input.name.replace(/ /g,"_"),
    fields: {},
    timestamp: new Date()
}];

if ("properties" in input){
    for (let key in input.properties) {
        output.payload[0].fields[key] = input.properties[key].value;
    }
} else{
    output.payload[0].fields["value"] = input.payload;
}

return output;

 

[Michel Kohanim]

@fisix,

Once registered, the only ways to deregister are:

1. You disable the module on ISY (in the dashboard)

2. SCE decommission at their site. I have rarely seen this happen without a customer request

What you need to look at is the dashboard. What's the status? Does it say: scanning/joining/scan complete (it found a meter that has join flag enabled for this ISY), or scanning/scan complete (no meter in sight), or scanning/rejoining (ISY thinks that it's still connected to the meter).

With kind regards,
Michel

[fisix]

Hi Michel - I think it's always the scanning/rejoining, then "Established" and then "Service Discovery" but no data, and the time is unsynchronized (I think it syncs if it's really talking).

The error log shows a lot of this:

3737708507	0	-100	[DHCP] state=RENEW
3737747699	0	-100	[DHCP] state=RENEW
3737757608	0	-170001	[ZBSEP] Reg. not complete 16
3737757608	0	-170001	[ZBSEP] Reg. not complete 16
3737757608	0	-170001	[ZBSEP] Reg. not complete 16
3737787215	0	-100	[DHCP] state=RENEW
3737800699	0	-170001	[AMI] Failed synching time with the meter. 
3737800699	0	-170001	[AMI] Failed getting Local Time
3737800834	0	-170001	[ZBSEP] Comm errors
3737800834	0	-170001	[ZBSEP] Comm errors
3737800875	0	-170001	[AMI] Failed synching time with the meter. 
3737800880	0	-170001	[AMI] Failed getting Local Time
3737801035	0	-170001	[ZBSEP] Comm errors
3737801035	0	-170001	[ZBSEP] Comm errors
3737801076	0	-170001	[AMI] Failed synching time with the meter. 
3737801081	0	-170001	[AMI] Failed getting Local Time

 

and then most recently:

3737930956	0	-170001	[Network] Established
3737930991	0	-170001	[AMI] Failed synching time with the meter. 
3737930996	0	-170001	[AMI] Failed getting Local Time
3737931382	0	-5	Start
3737931390	0	-100	No network connection
3737931445	0	-50010	mail.universal-devices.com
3737931480	0	-100	[DHCP] state=NOT STARTED
3738008702	0	-5	Start
3738008708	0	-170001	[Network] Established
3738015890	0	-170001	[UDSockets] HTTP:33 error:6
3738015895	0	-170001	[UDSockets] HTTP:33 error:6
3738015900	0	-170001	[UDSockets] HTTP:33 error:6
3738015905	0	-170001	[UDSockets] HTTP:33 error:6
3738015910	0	-170001	[UDSockets] HTTP:33 error:6
3738015915	0	-170001	[UDSockets] HTTP:33 error:6
3738015920	0	-170001	[UDSockets] HTTP:33 error:6
3738015925	0	-170001	[UDSockets] HTTP:33 error:6
3738015930	0	-170001	[UDSockets] HTTP:33 error:6
3738015935	0	-170001	[UDSockets] HTTP:33 error:6
3738015940	0	-170001	[UDSockets] HTTP:33 error:6
3738015945	0	-170001	[UDSockets] HTTP:33 error:6
3738015950	0	-170001	[UDSockets] HTTP:33 error:6
3738015955	0	-170001	[UDSockets] HTTP:33 error:6
3738015960	0	-170001	[UDSockets] HTTP:33 error:6
3738015965	0	-170001	[UDSockets] HTTP:33 error:6
3738015970	0	-170001	[UDSockets] HTTP:33 error:6
3738015975	0	-170001	[UDSockets] HTTP:33 error:6
3738015980	0	-170001	[UDSockets] HTTP:33 error:6
3738015985	0	-170001	[UDSockets] HTTP:33 error:6
3738015990	0	-170001	[UDSockets] HTTP:33 error:6
3738015995	0	-170001	[UDSockets] HTTP:33 error:6
3738016000	0	-170001	[UDSockets] HTTP:33 error:6
3738016005	0	-170001	[UDSockets] HTTP:33 error:6
3738020409	0	-5012	26

[Michel Kohanim]

Hi fisix,

Does the SCE console say that registration has failed? If so, ISY is trying to join but the meter is not allowing it. It could be that SCE unregistered your ISY.

In this case, you need to:

1. Soft Reset the Zigbee module on ISY (it's in the menu)

2. Redo the SCE registration 

With kind regards,
Michel

[fisix]

Thanks Michel,

It's a weird case.  I was getting good data for about 4 days, but in the middle of that the data stopped coming through.  I checked SCE's website and noticed that the status had changed to "registration failed".

I called them this morning and they're reopening the registration, and now the SCE website says that registration is pending again.  I don't think that's propagated out to my meter just yet, because the ISY isn't able to sync yet.

To me, this seems like a failure on their end - a time out of the registration process occurred even though I was successfully getting data, so either their back end munged something, or my meter didn't report back that it was communicating properly.  And, eventually, their back end timed out the registration.

If this process doesn't work, I'll do the soft reset.  However, I think they'll block the same EUID from registering again, even if there's a new install code.  Does the soft reset change both the EUID and the install code?

When I mentioned restarting from scratch, the SCE rep said it can take a few weeks to deregister on their end (ug).

In any event, thanks for the info, I'll update as to status when it changes.  I went ahead and moved the ISY back to the closest position to the meter just to eliminate potential collateral issues and to put it back in the position when it first started getting solid data.

[Michel Kohanim]

Hi fisix,

That's quite odd since there shouldn't be any registration attempts after ISY joins. It makes no sense.

In all cases YOU MUST do a soft reset because the registration process is going to use a different key.

With kind regards,

Michel

[fisix]

Good to know.  Just FYI - I had tried before to just re-register, but as long as the EUID is the same, their system blocks the re-register.  So far, it seems very hands on for them, because the only thing I can self-help is the initial device registration - after that, I'm at their mercy.  I can't delete or de-register devices.

So... the EUID for the Zigbee module... does that change after a soft reset?