Subscription Data Questions

[oratzes]

Hi,

 

Looking for some info on subscription data:

 

1.) Where can I find the definitions of the data structures and various codes that are being sent through the subscription?

2.) Am I correct that there is no timestamp being transmitted with the event subscription data? Can that be configured or changed? Really need a reference point of time when dealing with subscription events, especially since I am getting a blast of ~450 events the first time I attach a subscription. Not having a timestamp is making it impossible to compare prior subscription data to the new events

 

Thanks!

[evilpete]

Hi,

 

Looking for some info on subscription data:

 

1.) Where can I find the definitions of the data structures and various codes that are being sent through the subscription?

2.) Am I correct that there is no timestamp being transmitted with the event subscription data? Can that be configured or changed? Really need a reference point of time when dealing with subscription events, especially since I am getting a blast of ~450 events the first time I attach a subscription. Not having a timestamp is making it impossible to compare prior subscription data to the new events

 

Thanks!

 

 

have you looked http://www.universal-devices.com/isy-developers/

 

download & look at the Rest API docs

[PurdueGuy]

When you first subscribe, the blast is the current state of everything (every node, variables, program, etc).

You stay subscribed, then as changes occur, the ISY sends an update for only that item which changed.

[evilpete]

When you first subscribe, the blast is the current state of everything (every node, variables, program, etc).

You stay subscribed, then as changes occur, the ISY sends an update for only that item which changed.

 

 

... and unfortunately there's no marker, separating the "current state" blast from the following stream of live events

[PurdueGuy]

Why does it matter? What is your use case?

The latest message is always the latest state.

Are you attempting to build logic to know how long it has been since something happened, and then take some action?

 

You could connect, and until you don't get events for some configurable time (say 3 seconds?) then consider it all the initial state. After the timeout, consider the events that come to be occurring in real time, and you can timestamp them.

[evilpete]

Why does it matter? What is your use case?

The latest message is always the latest state.

Are you attempting to build logic to know how long it has been since something happened, and then take some action?

 

You could connect, and until you don't get events for some configurable time (say 3 seconds?) then consider it all the initial state. After the timeout, consider the events that come to be occurring in real time, and you can timestamp them.

 

if you have a program or app that is to take actions based to current events, you don't want it reacting events that happened hours previous or days previous.

 

A simple example might be an action that happens WHEN a door is opened

, as opposed to IF a door opened

[oratzes]

 

have you looked http://www.universal-devices.com/isy-developers/

 

download & look at the Rest API docs

 

First of all let me thank you for your work on github and your replies! Actually using your Python ISY library at the moment.

 

Let me explain the use case:

 

As there are more and more IoT system coming together I need a realtime integration of various data streams. My first use case is data visualization and analytics. Later one I want to derive and create new events to perform actions. The web resources feature in the ISY is nice but way too limiting. Just to integrate a Pentair Pool automation system or Push Notifications requires countless variables, network resources and does not give me the flexibility I need to do certain logic at scale.

 

This is a fairly large install with 150+ INSTEON devices, close to as many scenes, includes technologies like network attached systems that are not natively supported (TVs, Sonos), 40+ irrigation stations, well pumps, soon to be solar, multi zoned NEST (thermostats and smoke detectors), automatic gate, video cameras (SecuritySpy), ...

 

Needless to say there is no single place to go and get a status or data feed of what is going on. And especially the INSTEON devices seem to have a half live of less than a moon cycle (at least some do) and freeze, lockup, start beeping. Detecting when thing go wrong or not as intended is important.

 

I finally got around to setup an old Mac Mini (2010 model), replaced the HDDs with SSDs and added it to my home network as a worker node with the idea of leveraging python for much of the work, but combining it with state of the art realtime technologies as we use them for the likes of DevOps at scale. That means integrate and capture data in realtime and store in some sort of a database.

 

Here is the setup I put together in a matter of days:

 

Max Mini 2010 OSX Mavericks

supervisord (http://supervisord.org) to orchestrate the various software components and data feeds

InfluxDB (http://influxdb.com) as a realtime event database to stream all events from all technologies

ElasticSearch (http://www.elasticsearch.org) to index various pieces of data and make them all searchable

Grafana (http://grafana.org) as a light weight HTML5 visualization frontent

Python scripts to subscribe or pull events and post them into the InfluxDB backend

 

Here are a few screenshots to give you an idea:

 

 

 

 

And here is a code snippet for the current ISY event feed:

 

#!/usr/local/bin/python -u
__author__ = "Oliver Ratzesberger"

print "Starting InfluxDB JSON feed for Universal Devices ISY994"

import requests
import json
import sys
import os
import ConfigParser

from  ISY.IsyEvent import ISYEvent

config = ConfigParser.ConfigParser()
config.read(os.path.expanduser('~/home.cfg'))

isy_addr = config.get('isy', 'addr')
isy_user = config.get('isy', 'user')
isy_pass = config.get('isy', 'pass')
influx_addr = config.get('influxdb', 'addr')
influx_user = config.get('influxdb', 'user')
influx_pass = config.get('influxdb', 'pass')

def event_feed(*arg):

 data = arg[0]

 event = [{
   'name': 'isy',
   'columns': data.keys(),
   'points': [ data.values() ]
 }]

 print event

 try:
   r = requests.post(influx_addr + "?u=" + influx_user + "&p=" + influx_pass, data=json.dumps(event))
   print r
 except Exception:
   print 'Exception posting data to ' + influx_addr
   pass


server = ISYEvent()
server.subscribe(addr=isy_addr, userl=isy_user, userp=isy_pass)
server.set_process_func(event_feed, "")

try:
 print "Use Control-C to exit"
 server.events_loop()   #no return
except KeyboardInterrupt:
 print "Exiting"

[oratzes]

And a 'baby' realtime dashboard that currently just shows realtime power feed to the house an a selection of ISY events:

 

[oratzes]

Ultimately this will graph in realtime various temperatures, load levels, light levels, system activity, ....

 

Just a tiny prototype right now.

 

Not filtering any of the event data at the moment, even heartbeat messages are being loaded, that way I can have logic if systems go away.

 

Now back to my original question: Have bee trying to look at all the documents but had not much luck finding details about what all should come in the event subscription. Yes - I can reverse engineer most of it, but knowing what the structures should be, would help a lot. Must be looking in the wrong spot, because none of the documents I downloaded seem to contain any of that.

[evilpete]

what are you using to monitor your current home energy usage / amps ?

 

 

As for Insteon reliability,

I have 25 to 30 Insteon device is installed and I think the last year and a half of only seeing one device crash ( lockup ).

 

while it is not supported directly by the protocol it is possible to ping and count the retransmission / hops between the PLM and th Insteon node. Sadly this feature is not offered by the ISY (yet)

 

if you have any patches or enhancements for ISYlib please send them along

 

BTW : it would not be that hard to modify my python library to monitor / control multiple ISY devices

[evilpete]

Ultimately this will graph in realtime various temperatures, load levels, light levels, system activity, ....

 

Just a tiny prototype right now.

 

Not filtering any of the event data at the moment, even heartbeat messages are being loaded, that way I can have logic if systems go away.

 

Now back to my original question: Have bee trying to look at all the documents but had not much luck finding details about what all should come in the event subscription. Yes - I can reverse engineer most of it, but knowing what the structures should be, would help a lot. Must be looking in the wrong spot, because none of the documents I downloaded seem to contain any of that.

 

take a look ISYlib's internal _read_event() function, the code is ugly and needs to be rewritten but it I might give you some hints

 

https://github.com/evilpete/ISYlib-python/blob/dev/ISY/IsyClass.py#L409

[oratzes]

what are you using to monitor your current home energy usage / amps ?

 

 

As for Insteon reliability,

I have 25 to 30 Insteon device is installed and I think the last year and a half of only seeing one device crash ( lockup ).

 

while it is not supported directly by the protocol it is possible to ping and count the retransmission / hops between the PLM and th Insteon node. Sadly this feature is not offered by the ISY (yet)

 

if you have any patches or enhancements for ISYlib please send them along

 

BTW : it would not be that hard to modify my python library to monitor / control multiple ISY devices

 

Rainforest Eagle - polling it approx every 5 sec to get the latest demand numbers. Could be written into ISY variables, but had no need for that yet. Also not forwarding the kWh data just yet.

 

Messed up my ISYlib fork while playing with some of the test programs. Will definitely give you a pull request if there is anything meaningful I can contribute.

 

#!/usr/local/bin/python -u
__author__ = "Oliver Ratzesberger"

print "Starting InfluxDB JSON feed for Rainforest Eagle"

import requests
import json
import random
import time
import os
import ConfigParser

from RainEagle import Eagle, to_epoch_1970

config = ConfigParser.ConfigParser()
config.read(os.path.expanduser('~/home.cfg'))

eagle_addr  = config.get('raineagle', 'addr')
influx_addr = config.get('influxdb', 'addr')
influx_user = config.get('influxdb', 'user')
influx_pass = config.get('influxdb', 'pass')

eg = Eagle(debug=0, addr=eagle_addr)

while True:
 raindata= eg.get_device_data()

 idemand = raindata['InstantaneousDemand']

 multiplier = int(idemand['Multiplier'], 16)
 divisor = int(idemand['Divisor'], 16)
 demand = int(idemand['Demand'], 16)

 if demand > 0x7FFFFFFF: demand -= 0x100000000

 if multiplier == 0 : multiplier = 1

 if divisor == 0 : divisor = 1

 power = ((demand * multiplier) / float (divisor))*1000
 amps = power/240

 event = [{
   'name': 'power',
   'columns': [ 'whole_house_power', 'whole_house_amps' ],
   'points': [[ power, amps ]]
 }]

 print event

 try:
   r = requests.post(influx_addr + "?u=" + influx_user + "&p=" + influx_pass, data=json.dumps(event))
   print r
 except Exception:
   print 'Exception posting data to ' + influx_addr
   pass

 time.sleep(5)

[oratzes]

 

take a look ISYlib's internal _read_event() function, the code is ugly and needs to be rewritten but it I might give you some hints

 

https://github.com/evilpete/ISYlib-python/blob/dev/ISY/IsyClass.py#L409

 

Thank you! Done that - was my very first lesson.

 

For now I decided to take all data AS-IS - given that InfluxDB is schema less and will happily ingest all of it. That way I can analyze the data better and see what is happing over longer periods of time.

[evilpete]

what are you using to monitor your current home energy usage / amps ?

 

 

Rainforest Eagle - polling it approx every 5 sec to get the latest demand numbers. Could be written into ISY variables, but had no need for that yet. Also not forwarding the kWh data just yet.

 

 

I've started on Rainforest Eagle library

 

https://github.com/evilpete/RainEagle

 

it is not as advanced as the ISY library but it exposes the undocumented web API as well as the documented socket based API

[oratzes]

I've started on Rainforest Eagle library

 

https://github.com/evilpete/RainEagle

 

it is not as advanced as the ISY library but it exposes the undocumented web API as well as the documented socket based API

 

I know - its what I am using - see feeder code above

 

from RainEagle import Eagle

[oratzes]

Here is an example of what you see inside a minute. A few lights go on, followed by some dimming and all of a sudden ERROR start hitting. This is the type of behavior I need to better understand and see if it contributes to certain switches locking up every few weeks.

 

[Michel Kohanim]

Hi oratzes,

 

Can you please let me know what you need from ISY perspective? Information on events and what they mean can be found in our WSDK. Beyond that, what else are you looking for?

 

With kind regards,

Michel

[evilpete]

Here is an example of what you see inside a minute. A few lights go on, followed by some dimming and all of a sudden ERROR start hitting. This is the type of behavior I need to better understand and see if it contributes to certain switches locking up every few weeks.

 

[attachment=0]Screen Shot 2014-06-26 at 10.04.26 AM.png[/attachment]

 

my suggestion is to start documenting all details till patterns emerge.

 

 

some ideas :

 

what type of failures? do they stop operating remotely, locally or both ?

 

Are multiple devices failing at the sametime ?

 

are they the same devices is it random

 

are they all on the same circuit? sub pannel ? power phase ?

 

is it a network access problem? ( eg: a intermediate device failing to relay commands

 

do you have a method to monitor the power quality ? Many UPS have the ability to do this as a feature of their software.

 

I have also found UPSs are the worst offenders, inducing a lot of low voltage noise ( I have all 4 of my home UPSs on filters )

[oratzes]

Hi oratzes,

 

Can you please let me know what you need from ISY perspective? Information on events and what they mean can be found in our WSDK. Beyond that, what else are you looking for?

 

With kind regards,

Michel

 

Two things:

 

1.) Can you post the link to the document/page that has the subscription event information details? I keep searching and I find all kinds of SOAP and REST documentation but nothing about what the event data contains that is being sent to a subscriber. Sorry if I am missing the obvious

 

2.) Is there any way to separate startup communication of a subscription from realtime notifications? Subscriptions are not super stable - meaning, they drop a few times in any 24h period and require a re-connect. When you are doing realtime event processing, the startup messages get in the way of actually seeing some real patterns. Especially since I am getting more then 470 of them each time I connect. Any flag or counter/number that helps separate the initial dump of all statuses from the ongoing event stream would be extremely helpful.

 

Thanks!

 

Oliver

[io_guy]

Why does it matter? What is your use case?

The latest message is always the latest state.

Are you attempting to build logic to know how long it has been since something happened, and then take some action?

 

You could connect, and until you don't get events for some configurable time (say 3 seconds?) then consider it all the initial state. After the timeout, consider the events that come to be occurring in real time, and you can timestamp them.

 

if you have a program or app that is to take actions based to current events, you don't want it reacting events that happened hours previous or days previous.

 

A simple example might be an action that happens WHEN a door is opened

, as opposed to IF a door opened

I don't see the issue here. On connection, your program's code should just ignore the first value change, since it's current status. I have a ton of programs that do this.

You could also pull original values via REST and compare.

There's 100 ways to skin this cat.

[Michel Kohanim]

Hi Oliver,

 

The WSDK documentation on page 37:

http://www.universal-devices.com/develo ... -4.2.5.zip

 

With kind regards,

Michel

[oratzes]

Found it thank you!

 

Too many files and I kept opening and searching the wrong ones and the wrong version as it turns out too.

 

Maybe you can convert this to wiki or website content at some point in time. That way even google would find it.

[oratzes]

Why does it matter? What is your use case?

The latest message is always the latest state.

Are you attempting to build logic to know how long it has been since something happened, and then take some action?

 

You could connect, and until you don't get events for some configurable time (say 3 seconds?) then consider it all the initial state. After the timeout, consider the events that come to be occurring in real time, and you can timestamp them.

 

if you have a program or app that is to take actions based to current events, you don't want it reacting events that happened hours previous or days previous.

 

A simple example might be an action that happens WHEN a door is opened

, as opposed to IF a door opened

I don't see the issue here. On connection, your program's code should just ignore the first value change, since it's current status. I have a ton of programs that do this.

You could also pull original values via REST and compare.

There's 100 ways to skin this cat.

 

If you do realtime event processing it makes a WORLD of a difference. Sure you can write more code and filter, but the amount of extra programming is in no relation to a simpler and cleaner approach that simply allows you to differentiate between startup and true realtime events and the true timestamp would make it much more precise to record WHEN an event occurred. With the current setup you need to read the log AND subscribe, dedupe and sessionize the data plus deal with all the edge cases.

 

The architecture would benefit from the simple addition of a timestamp and those that don't need it could simply ignore it.

 

Actually found something in the protocol that might help with part of the problem. Need to look at the raw data.

[PurdueGuy]

The architecture would benefit from the simple addition of a timestamp and those that don't need it could simply ignore it.

As far as I have seen, ISY doesn't keep track of timestamps for the current state of devices. It has timestamps for programs and variables, since you can write conditions for them, but not device/node status.