oratzes

if you have a program or app that is to take actions based to current events, you don't want it reacting events that happened hours previous or days previous. A simple example might be an action that happens WHEN a door is opened , as opposed to IF a door opened I don't see the issue here. On connection, your program's code should just ignore the first value change, since it's current status. I have a ton of programs that do this. You could also pull original values via REST and compare. There's 100 ways to skin this cat. If you do realtime event processing it makes a WORLD of a difference. Sure you can write more code and filter, but the amount of extra programming is in no relation to a simpler and cleaner approach that simply allows you to differentiate between startup and true realtime events and the true timestamp would make it much more precise to record WHEN an event occurred. With the current setup you need to read the log AND subscribe, dedupe and sessionize the data plus deal with all the edge cases. The architecture would benefit from the simple addition of a timestamp and those that don't need it could simply ignore it. Actually found something in the protocol that might help with part of the problem. Need to look at the raw data.

Found it thank you! Too many files and I kept opening and searching the wrong ones and the wrong version as it turns out too. Maybe you can convert this to wiki or website content at some point in time. That way even google would find it.

Two things: 1.) Can you post the link to the document/page that has the subscription event information details? I keep searching and I find all kinds of SOAP and REST documentation but nothing about what the event data contains that is being sent to a subscriber. Sorry if I am missing the obvious 2.) Is there any way to separate startup communication of a subscription from realtime notifications? Subscriptions are not super stable - meaning, they drop a few times in any 24h period and require a re-connect. When you are doing realtime event processing, the startup messages get in the way of actually seeing some real patterns. Especially since I am getting more then 470 of them each time I connect. Any flag or counter/number that helps separate the initial dump of all statuses from the ongoing event stream would be extremely helpful. Thanks! Oliver

Here is an example of what you see inside a minute. A few lights go on, followed by some dimming and all of a sudden ERROR start hitting. This is the type of behavior I need to better understand and see if it contributes to certain switches locking up every few weeks.

I know - its what I am using - see feeder code above from RainEagle import Eagle

Thank you! Done that - was my very first lesson. For now I decided to take all data AS-IS - given that InfluxDB is schema less and will happily ingest all of it. That way I can analyze the data better and see what is happing over longer periods of time.

Rainforest Eagle - polling it approx every 5 sec to get the latest demand numbers. Could be written into ISY variables, but had no need for that yet. Also not forwarding the kWh data just yet. Messed up my ISYlib fork while playing with some of the test programs. Will definitely give you a pull request if there is anything meaningful I can contribute. #!/usr/local/bin/python -u __author__ = "Oliver Ratzesberger" print "Starting InfluxDB JSON feed for Rainforest Eagle" import requests import json import random import time import os import ConfigParser from RainEagle import Eagle, to_epoch_1970 config = ConfigParser.ConfigParser() config.read(os.path.expanduser('~/home.cfg')) eagle_addr = config.get('raineagle', 'addr') influx_addr = config.get('influxdb', 'addr') influx_user = config.get('influxdb', 'user') influx_pass = config.get('influxdb', 'pass') eg = Eagle(debug=0, addr=eagle_addr) while True: raindata= eg.get_device_data() idemand = raindata['InstantaneousDemand'] multiplier = int(idemand['Multiplier'], 16) divisor = int(idemand['Divisor'], 16) demand = int(idemand['Demand'], 16) if demand > 0x7FFFFFFF: demand -= 0x100000000 if multiplier == 0 : multiplier = 1 if divisor == 0 : divisor = 1 power = ((demand * multiplier) / float (divisor))*1000 amps = power/240 event = [{ 'name': 'power', 'columns': [ 'whole_house_power', 'whole_house_amps' ], 'points': [[ power, amps ]] }] print event try: r = requests.post(influx_addr + "?u=" + influx_user + "&p=" + influx_pass, data=json.dumps(event)) print r except Exception: print 'Exception posting data to ' + influx_addr pass time.sleep(5)

Ultimately this will graph in realtime various temperatures, load levels, light levels, system activity, .... Just a tiny prototype right now. Not filtering any of the event data at the moment, even heartbeat messages are being loaded, that way I can have logic if systems go away. Now back to my original question: Have bee trying to look at all the documents but had not much luck finding details about what all should come in the event subscription. Yes - I can reverse engineer most of it, but knowing what the structures should be, would help a lot. Must be looking in the wrong spot, because none of the documents I downloaded seem to contain any of that.

And a 'baby' realtime dashboard that currently just shows realtime power feed to the house an a selection of ISY events:

First of all let me thank you for your work on github and your replies! Actually using your Python ISY library at the moment. Let me explain the use case: As there are more and more IoT system coming together I need a realtime integration of various data streams. My first use case is data visualization and analytics. Later one I want to derive and create new events to perform actions. The web resources feature in the ISY is nice but way too limiting. Just to integrate a Pentair Pool automation system or Push Notifications requires countless variables, network resources and does not give me the flexibility I need to do certain logic at scale. This is a fairly large install with 150+ INSTEON devices, close to as many scenes, includes technologies like network attached systems that are not natively supported (TVs, Sonos), 40+ irrigation stations, well pumps, soon to be solar, multi zoned NEST (thermostats and smoke detectors), automatic gate, video cameras (SecuritySpy), ... Needless to say there is no single place to go and get a status or data feed of what is going on. And especially the INSTEON devices seem to have a half live of less than a moon cycle (at least some do) and freeze, lockup, start beeping. Detecting when thing go wrong or not as intended is important. I finally got around to setup an old Mac Mini (2010 model), replaced the HDDs with SSDs and added it to my home network as a worker node with the idea of leveraging python for much of the work, but combining it with state of the art realtime technologies as we use them for the likes of DevOps at scale. That means integrate and capture data in realtime and store in some sort of a database. Here is the setup I put together in a matter of days: Max Mini 2010 OSX Mavericks supervisord (http://supervisord.org) to orchestrate the various software components and data feeds InfluxDB (http://influxdb.com) as a realtime event database to stream all events from all technologies ElasticSearch (http://www.elasticsearch.org) to index various pieces of data and make them all searchable Grafana (http://grafana.org) as a light weight HTML5 visualization frontent Python scripts to subscribe or pull events and post them into the InfluxDB backend Here are a few screenshots to give you an idea: And here is a code snippet for the current ISY event feed: #!/usr/local/bin/python -u __author__ = "Oliver Ratzesberger" print "Starting InfluxDB JSON feed for Universal Devices ISY994" import requests import json import sys import os import ConfigParser from ISY.IsyEvent import ISYEvent config = ConfigParser.ConfigParser() config.read(os.path.expanduser('~/home.cfg')) isy_addr = config.get('isy', 'addr') isy_user = config.get('isy', 'user') isy_pass = config.get('isy', 'pass') influx_addr = config.get('influxdb', 'addr') influx_user = config.get('influxdb', 'user') influx_pass = config.get('influxdb', 'pass') def event_feed(*arg): data = arg[0] event = [{ 'name': 'isy', 'columns': data.keys(), 'points': [ data.values() ] }] print event try: r = requests.post(influx_addr + "?u=" + influx_user + "&p=" + influx_pass, data=json.dumps(event)) print r except Exception: print 'Exception posting data to ' + influx_addr pass server = ISYEvent() server.subscribe(addr=isy_addr, userl=isy_user, userp=isy_pass) server.set_process_func(event_feed, "") try: print "Use Control-C to exit" server.events_loop() #no return except KeyboardInterrupt: print "Exiting"

Hi, Looking for some info on subscription data: 1.) Where can I find the definitions of the data structures and various codes that are being sent through the subscription? 2.) Am I correct that there is no timestamp being transmitted with the event subscription data? Can that be configured or changed? Really need a reference point of time when dealing with subscription events, especially since I am getting a blast of ~450 events the first time I attach a subscription. Not having a timestamp is making it impossible to compare prior subscription data to the new events Thanks!

I understand. Just trying to get some data in the meantime. I am also playing with an Opensource realtime metrics dashboard and would like to stream all kinds of data (including power) to its realtime graphs. That way I can see data in realtime.

Hi evilpete, I have been looking into a similar integration. First of all thank you for the python work you have done so far. Rather than pulling the EAGLE at preset intervals I want to 'listen' to the stream that it sends to bidgely. I am waiting for my firmware to get updated and want to setup a lightweight POST server one a Mac Mini in python, that consumes the upload data stream and forwards it to bidgely - as a proxy. With every data packet it gets it should update a view variables on the ISY to reflect the most current energy consumption. FYI - Doing all of this as the ISY - meter integration with San Diego Gas & Electric does not seem to work. Have you done anything similar? Thanks! Oliver

Found the problem with the help from support. The issues was that the local machines hostname was out of sync and missing in the local hosts file. In my case the installation of OSX Mavericks Server (App from the App Store) took over control of the hostname without keeping the sharing hostname and the hostile in sync. If you are running OSX Mavericks + the OSX server App, make sure you have the name of the local machine in sync in all three places (see screenshot). Hope this helps others!

Since the very first day I purchased the ISY 994 PRO I have never been able to start the Dashboard or Admin Console from the various download links here on the site. Aside from all the Java issues and security overrides it requires to make these Apps even start up on a Mac (10.9 Mavericks, Java 7 Update 55), I have never managed to get anything but a NOT FOUND error in the initial Finder window that opens. Entering the fixed IP address or DNS name I setup for the ISY always leads to more INVALID URL error message. My system is a Mac Mini setup as a OSX Server (running DHCP, DNS amongst other services), an Airport Extreme for NAT. My client machine is a MacBook Air also on Mavericks. My ISY is at address 10.0.1.199 and has a nice DNS name. The ISY 994 PRO is up and running and fully accessible through a web browser. Until now I have been using http://10.0.1.199/admin to get to the admin console and was able to do most of what I needed. Just learned that I must use the dashboard for various settings tasks. Here is where I am stuck. I can download the java apps for dashboard and admin console from the site here and can accept the various security exceptions to even get them to run, but the Finder always comes up empty, not matter what I try. Clicking the Add button always leads to an invalid URL error message, even when I copy paste the exact URL from the browser into it. What am I missing? (I am very computer literate, come from software development and manage my own home network, VPN, DHCP, DNS, but I am stuck with a little DYI home automation device...)