Energy Monitoring: Circuits, Plugs, Interface

[MWareman]

I'm thinking of a program that posts to a rest api every 5 mins with the current value of each zwave energy monitor. Not possible until we get variable substitution in network resources, without resorting to external scripting.

 

That being said, there is no reason an external script couldn't obtain power data on a schedule via ISYs REST api, then post to the power monitoring site.

[ccclapp]

Thanks

I don't feel quite as dumb now

[MaddBomber83]

[ccclapp]

Hi madbomber83

 

This is great! Thank you so much!!

 

I am the least expert person in the group. However I will be speaking to "the Wentworth student" today and also to Xathros. Either may possibly reply to you directly before then, but in any event your post will be part of our conversations.

 

Thanks for taking the initiative to get us to this next step!!

[Xathros]

Use an ISY program to issue the network resource. Call the program RunThen from the /REST interface with your javascript?

 

-Xathros

 

EDIT: Nevermind. I misunderstood the issue there.

[munroe7]

@maddbomber83- Where is isy's logger interface?

[MaddBomber83]

The guide, ish, here shows how to do this.

 

Sorry to Munroe7, I did not see your question. ISY has a data logger under Tools, but that is not what we are using here.

 

Instead we are using ISY's ability to append values to the end of a CSV file that is storred on a web accessable address on the ISY itself.

 

An example is a notification set up like this:

 

Subject: @webpage:@append:HVAC.csv

Body: ${sys.date},${sys.sunrise24},${sys.sunset24},${mod.weather.temp.high},${mod.weather.temp.low},${var.1.21},${var.1.23},${var.1.9},${var.1.14},${var.1.15},${var.1.17},${var.1.18},${var.1.20},${var.1.33}

 

 

I then call this notification in a program once a day. All those ${var.1.##} are run times for my HVAC (Off, Fan, Cooling, Heating).

 

It stores this CSV on the ISY at http://isy/user/web/HVAC.CSV.

 

I can then access this file remotely or locally. I just have it set as a data source in excel.

 

This is what that CSV looks like:

Date	Sunrise	Sunset	High Temperature	Low Temperature	Off	Off_D	Fan	Fan_D	Cool	Cool_D	Heat	Heat_D	Err
5/7/2014	6:58:33	20:35:19	73 °F	63 °F	305	7	0	0	0	0	0	0	23
5/7/2014	6:58:33	20:35:19	78 °F	78 °F	386	-1053	0	0	0	0	0	0	6
5/8/2014	6:57:43	20:36:01	66 °F	66 °F	1434	1046	0	0	0	0	0	0	23
5/9/2014	6:56:53	20:36:43	66 °F	66 °F	1437	1	0	0	0	0	0	0	23
5/10/2014	6:56:05	20:37:25	71 °F	71 °F	864	-574	573	572	0	0	0	0	23
5/11/2014	6:55:18	20:38:06	77 °F	77 °F	611	-254	853	279	0	0	0	0	24

 

You can do the same thing with power monitoring. I did find it useful to throttle the number of updates for power monitoring by using 2 programs where one of them had a last run flag on it.

[ccclapp]

Thanks for that description, maddbomber83. Your work on this is a big help!! I wish I was more advanced in these areas of ISY and general scripting knowledge. As the OP on this thread, I should be the one figuring all of this out, but I haven't. I will try to follow your steps and see if I can make basic sense of it and then return with questions.

 

I believe this topic is or will be VERY interesting to the wider ISY user base, i.e how to get plug/device energy data from ISY and either use it in ISY directly, or upload it to great energy monitoring sites. This: 1) enables ISY actions/controls triggered by very specific (at the plug level) energy use and 2) allows for whole house AND specific plug energy data with pretty charts and graphs showing current, day, week, month, year views using the widgets on the referenced monitoring sites. For whole-house monitoring there are inexpensive Z-wave CT clamp units that attach to one's circuit panel avoid the cost and complexity of a full-blown system like the GEM, etc. This can be one of the few areas where ISY shows its strengths on a beautiful web interface at a low cost, that will surely impress your friends and family, helping justify all the time we spend on this "hobby"

 

Monroe7 has been unavailable to assist me with this, but I hope to get him back in the fold... in the meantime, all input/progress from others is greatly appreciated, and as stated, I expect this will become a meaningful feature of ISY for many users...

 

Thanks

[Scyto]

I know this an old thread. I have ISY, GEM, DB and zwave energy monitors.

 

Did you ever find a way to intelligently combine it all into one beautiful mobile app/ portal?

[ccclapp]

hi

 

i have not done it via isy but am about to embark on the same by integrating a vera into my isy config as descrined in this thread

 

http://forum.micasaverde.com/index.php?topic=9998.new;topicseen#new

 

( actually as you will see this does not require a vera for the gem w db for meter monitoring but does if you want plugs too)

 

also here is an isy only thread, agian without plugs

 

http://forum.universal-devices.com/topic/13801-dash-box-state-variables-now-supported/page-2#

 

based on the info in this thread ( the one you just posted to) i expect plugs can be done via isy but dont know scripting to implement myself). hopefully someone smarter than i will pull it together.

 

in my case im adding a vera to add plug data and push both ( plug/gem) to SEG as described in the top thread link. i am also using the isy plugin for the vera to link the vera to the isy and see/control all isy devices/programs but have a better UI, 2-way communication, which vera can do, and use vera plugins eg energy, sonos and others.

 

maybe w isy v5.xx i wont need vera, but i do now to overcome the above isy limitations (at least for a nonscripter like me)

 

lut us know where you end up. good luck

[ccclapp]

HI, here is a brief update and some info that might possibly help us get to the next step...

 

SUMMARY TO DATE

• Goal:  To be able to monitor smart plugs and elect panel circuits and send energy streams to SEG or other monitoring site (on SEG the streams can be graphed, merged, subtracted, etc, etc)

 

• Status:
  • On ISY Panel/circuit monitoring can be done with the GEM (possibly requiring the Dash Box add-on (see 2nd link in post directly above))
  • On ISY plug monitoring has not been accomplished (although Maddbomber83 has been able to chart plug data to local CSV file).
  • On Vera all of this can be done (see 1st link in post directly above), but then we have to use a Vera 
• Workaround:
  • For me, having BOTH panel and plug info is important.  Too many high power devices are on the same circuit to only monitor circuits, especially AV and PC (whole-house AV set up in central location on single circuit drawing 450-650W and PC/Servers/office additionally drawing  300-450W)
  • I split my Zwave network into 2:  1 for normal on ISY and 2nd on Vera for power monitoring (and a few plugins like Sonos).   NOTE:  Vera has an ISY plugin, which should allow me to have all zwave on ISY and see all devices on Vera and link to SEG from Vera.  I have confirmed the plugin picks up the ISY devices on Vera, but not fully tested this approach.
  • My strong preference would be to keep it all on ISY if we can get ISY to report to SEG.
• Hope:
  • The Vera script to report to SEG is quite good and easy to set up.  The script text is fairly brief and seems straightforward.  I am hoping someone who can script on ISY might be able to adapt it for our use [HUGE and RUDE HINT, HINT / BEG, BEG], so I am copying and linking it here with that hope 
  • To make the script work (on Vera) one only (i) edits it for their own SEG Site Token ID #, (ii) edits the SEG Devices list by a) adding the desired data stream name from you SEG Streams (e.g. the stream you add/create to receive the energy data for your smart plug, b ) edit for the applicable Zwave device ID # (e.g. your smart plug), and confirm or edit the Zwave "service ID" and "service variable" found in the device (smart plug) zwave properties.  I have highlighted these in the script below.
  • Hopefully the approach taken in this brief Vera script is sufficiently applicable/adaptable to ISY to get us to the finish line (or at least a little further down the field).
  • Unfortunately I don't know the ins/outs of ISY scripting, variables and network resources to tackle this myself.  However, I am happy to assist with the SEG and Zwave parts. 
  • Here is the link and script:  http://forum.micasaverde.com/index.php/topic,9998.msg175678.html#msg175678
  • NOTE:  All credit for this script does to @Guessed on the Vera forum.  I had zero to do with it

local http = require('socket.http')
http.TIMEOUT = 5

local SEG_SITE = "<yourSEGSiteIdHere>"
local SEG_URL = "http://api.smartenergygroups.com/api_sites/stream"
local segLog = function (text) luup.log('SEG Logger: ' .. (text or "empty")) end

SEG_DEVICES = {
    {node='vera', stream='t_upstairs',   deviceId=335, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_downstairs', deviceId=384, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_outside',    deviceId=318, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"}
  }

local function findStream(deviceId, serviceId, serviceVar)
  -- TODO: Change from a simple scan to something that'll support more entries efficiently.
  for k, v in pairs(SEG_DEVICES) do
    if (v.deviceId == deviceId and
        v.serviceId == serviceId and
        v.serviceVar == serviceVar) then
      return v.node, v.stream
    end
  end

  return nil, nil
end

local function initWatch()
  for k, v in pairs(SEG_DEVICES) do
    luup.variable_watch('segWatch', v.serviceId, v.serviceVar, v.deviceId)
  end
end

function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
  local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)

  if (nodeName == nil or streamName == nil) then
    segLog(string.format("Node not found for %s,%s,%s", deviceId, serviceId, serviceVar))
    return
  end

  -- TODO: Encoding
  segData = string.format("(site %s (node %s ? (%s %s)))", SEG_SITE, nodeName, streamName, newValue)
  segLog(segData)

  result, status = http.request(SEG_URL, segData)
end

initWatch()

[Michel Kohanim]

Hi ccclapp,

 

I am not clear as to the requirement. Perhaps you could post the actual requirements here:

http://forum.universal-devices.com/forum/6-product-requests/

 

With kind regards,

Michel

[Scyto]

FYI cclap, I asked Ben at Brultech if they would look at pulling z-wave power usage data from the ISY into the DashBox.  He seemed interested but no commitment. Link

[MWareman]

FYI cclap, I asked Ben at Brultech if they would look at pulling z-wave power usage data from the ISY into the DashBox. He seemed interested but no commitment. Link

That would be *awesome*

[ccclapp]

Hi Folks

I'm dredging this one up again in hopes of figuring it out...

 

PRIOR SUMMARY:

-- Goal: Monitor energy use of BOTH plugs (insteon/zwave) and circuits (GEM or other appropriate device at the house panel) and send that data up to Smart Energy Groups (SEG) for monitoring/reporting/graphing, etc

-- Status as of last post 2/2015:

- Circuit data can be transmitted to SEG via GEM with DASH box (does not involve ISY)

- Some progress by MaddBomber83 in creating a .csv file through the ISYweb address in hopes of automating transmission to up to SEG

- Plug data can be transmitted to SEG using a convoluted workaround requiring adding a Vera to the mix

- Sycto initiated discussion with Brultech in a thread on their site of getting ISY plug data via the ISY  RESTful interface into the DASH and then up to SEG. 

 

​Conclusion of the above: Getting ISY device energy data (plugs) out of ISY and up to SEG has not been solved.  

 

RESUMING THE DISCUSSION:

1. Over the past year, have any of you made further progress in this?
2. Does v5.x ISY firmware make this more attainable now?
3. Does the ISY Portal help in any way?
4. Are there any 3rd party tools now available which can help?
5. Would introducing a RPi enable this and if so, does anyone know the steps?

Bottom, line, is there still interest in being able to monitor ISY device energy data on a site like SEG.

 

LINKS FOR CONVENIENCE:

- SEG:  https://smartenergygroups.com/

- Thread at Brultech: http://www.brultech.com/community/viewtopic.php?f=33&t=1281

- Brultech: http://www.brultech.com/

 

Thanks in advance for any interest/help/guidance!!

[Teken]

Hello ccclapp,

 

Not sure if you were aware one method I use for local energy monitoring is with the use of a specially modified UL / cUL plug. This plug is connected to a Micro CT which is then attached to one of 32 GEM channels for tracking and monitoring.

 

I have been using this method exactly for the same reason you have. As I have a few high current devices that sometimes are called to duty and I like to know all the energy metrics from inrush current, run time, duty cycle, etc.

 

 

 

[ccclapp]

Thanks Taken and glad to have your input on this thread!  I am aware you do that.  In my case I have many zwave/insteon smart plugs and would love to simply tap the data ISY has on them, instead of wiring to CT's back at the panel.  In my case the level of precision is not critical and I have no doubt your solution is far more precise/robust.  It just seems counter-intuitive that there doesn't seem to be a reasonable way to get the ISY-device energy data to SEG, etc.  

 

I'm about to link to some other 3rd-party ISY addon guys on this forum to see if they might be interested in helping with this...

 

Thanks 

[Teken]

Thanks Taken and glad to have your input on this thread!  I am aware you do that.  In my case I have many zwave/insteon smart plugs and would love to simply tap the data ISY has on them, instead of wiring to CT's back at the panel.  In my case the level of precision is not critical and I have no doubt your solution is far more precise/robust.  It just seems counter-intuitive that there doesn't seem to be a reasonable way to get the ISY-device energy data to SEG, etc.  

 

I'm about to link to some other 3rd-party ISY addon guys on this forum to see if they might be interested in helping with this...

 

Thanks 

 

I believe (IF) and when Ben at Brultech has cleared off the 60 plus things I've asked of him for the Dash Box (DB). The suggestion offered her and on the Brultech forum about supporting Z-Wave data extraction may move up the ladder for support.

 

As I understand it the over all goal and intent is to pull the data from the 994 Series Controller and in turn this data would be ingested by the DB.

 

The DB presently can only relay that same data to SEG.

 

The long term goal is to have the DB have the ability to direct these metrics to any location whether it be local or cloud hosted. As an aside the Brultech team is also waiting for UDI to issue a stable and solid 5.XX RC for public consumption. At this point nobody knows when a 5.XX RC will come to be.

 

Thus, the chicken and egg situation for some of these feature support requests. To be clear the 5.XX RC is only pertaining to the GEM supporting nodes and has nothing to do with Z-Wave support.

 

Again, to be crystal clear (IF) this was to be supported the DB would be doing all the heavy lifting in extracting the Z-Wave data from 994 Series Controller whether it be via the REST, polling, etc. So essentially if the data is in the 994 Series Controller once the DB is updated to grab said data the system would relay that same data to what ever data server of your choice.

 

Hope this gives you some insight and things to consider.

 

Lastly, on a personal level I can appreciate the long journey you have endured. I have been pushing all aspects of energy monitoring / management on many fronts for many years. Its safe to say with out the endless support and guidance of Brultech lots of the things you have read about would never have materialized.

 

Specifically you and many others did not want to give up Z-Wave in the 994 Series Controller. Because of this I initiated the idea of having the DB being able to send state variables from the GEM directly to the 994 Series Controller. This in essence allowed everyone to have the best of both worlds.

 

X-10, Insteon, Z-Wave support  . . . 

[ccclapp]

Again, to be crystal clear (IF) this was to be supported the DB would be doing all the heavy lifting in extracting the Z-Wave data from 994 Series Controller whether it be via the REST, polling, etc. So essentially if the data is in the 994 Series Controller once the DB is updated to grab said data the system would relay that same data to what ever data server of your choice.

 

Yes, that would be an excellent solution.  There may be other approaches for those without a DASH, e.g. RPi or what maddbomber was trying to do, but I am lucky to have a DASH

.

 

Lastly, on a personal level I can appreciate the long journey you have endured. I have been pushing all aspects of energy monitoring / management on many fronts for many years. Its safe to say with out the endless support and guidance of Brultech lots of the things you have read about would never have materialized.

 

Specifically you and many others did not want to give up Z-Wave in the 994 Series Controller. Because of this I initiated the idea of having the DB being able to send state variables from the GEM directly to the 994 Series Controller. This in essence allowed everyone to have the best of both worlds.

 

X-10, Insteon, Z-Wave support  . . . 

 

Yes, and thanks for your initiative in all of this and for helping carry that message forward! 

[Teken]

Back on topic: You currently have some Z-Wave plugs and switches that monitor energy correct? My question is can the 994 Series Controller presently track these values in state variables?

 

Meaning if you have a plug that says the TV is consuming 100 watts. Can you create a state variable to capture this 100 watts?

[ccclapp]

Back on topic: You currently have some Z-Wave plugs and switches that monitor energy correct? My question is can the 994 Series Controller presently track these values in state variables?

 

Meaning if you have a plug that says the TV is consuming 100 watts. Can you create a state variable to capture this 100 watts?

 

I should know this but dont know it.  I will try to find out.  I am on v4.x if that is relevant to your question.

[ccclapp]

I should know this but dont know it.  I will try to find out.  I am on v4.x if that is relevant to your question.

 

I guess I dont know how to link a state variable to a device??  After that I can try it with a zwave energy device to confirm it works w zwave...

[Teken]

I should know this but dont know it.  I will try to find out.  I am on v4.x if that is relevant to your question.

 

Well, if you determine that any Z-Wave device you have on hand that pertains to energy readings can in fact be stored as a *State Variable* I can offer you a interim solution that will allow you to track and ingest that same data to SEG.

[ccclapp]

Well, if you determine that any Z-Wave device you have on hand that pertains to energy readings can in fact be stored as a *State Variable* I can offer you a interim solution that will allow you to track and ingest that same data to SEG.

 

Out posts crossed in the cloud.  Please see my post, 1 above yours

[Teken]

I guess I dont know how to link a state variable to a device??  After that I can try it with a zwave energy device to confirm it works w zwave...

 

Your first task is to pose this question to UDI and the forum members with the particulars of your Z-Wave device. See if anyone has been able to take the various nodes from Z-Wave device(s) and transpose them into *State Variables*.

 

If that isn't presently supported then obviously you will need to wait for said support. If on the other hand this information can be scraped from the 994 Series Controller from what ever method that allows the state variable to be updated.

 

Then, you will have a nice aggregation solution waiting for you . . .