GCM / FCM

[G W]

I'd just like to point out that bandwidth should not be a problem. Each even only need send a few bytes to the portal. The portal should know what the data means.

 

Best regards,

Gary Funk

[mwester]

Bandwidth, and in particular limits, include packet overhead - so the few bytes blow up to a lot more by the time the event is wrapped in the XML tags by the ISY, and the TCP  and IP headers are added.  So filtering is absolutely necessary.

[giesen]

Here's what I would envision as the ideal scenario (James feel free to disagree with me/correct me):

 

1. ISY sends updates to ISY Portal for each state change (devices, scenes, variables, programs to start).

 

2. ISY Portal has an API for mobile app developers to hook into to subscribe to events from a particular ISY for their user. This can either be a continuous subscription socket, or connection initiated from ISY Portal Server on each event (ie REST call). Each mobile app vendor would run their own server that would subscribe to events from the ISY Portal.

 

3. Mobile App Vendor server submits the updates/notifications to the appropriate cloud messaging service (GCM/FCM/APNS) depending on what platform the user is running their app on.

 

4. Cloud messaging server pushes updates/notifications to app running on user's mobile device.

 

 

 

For those concerned about the volume of data transferred, if an efficient binary protocol is used (ie. not XML), 100 bytes per event would probably be more than generous. Assuming a very high 10000 events per day, that would be 1MB per day, or about 30 MB per month. Unless your ISY is connected to a 3G device in Iran without a roaming plan, that shouldn't be a concern for most people.

 

On the mobile device side, I believe Agave currently maintains an open socket with a subscription, so it's already receiving all the updates from an ISY, but with a much less efficient protocol, and as such is sucking up much more data (and no one is complaining about how much data Agave is using). What people should be complaining about, is that running a background process and keeping a socket open is sucking up BATTERY. Moving updates to being push-based rather than keeping a socket open will give longer battery life on your mobile device, and use less data than Agave is currently using. On mobile, battery life is everything.

 

Also, running background processes and keeping an open socket is not even really an option on IOS devices. So UDI implementing an API that mobile app developers solves the problems for all mobile app developers, regardless of whether they're developing on IOS, Android, Windows Mobile, Amazon Kindle Fire, Windows Desktop, Mac Desktop, etc.

 

What James wants will make for a better user experience, may stimulate competition for better mobile apps, and in general make cranky people like me happy

Edited October 8, 2016 by giesen

[giesen]

Bandwidth, and in particular limits, include packet overhead - so the few bytes blow up to a lot more by the time the event is wrapped in the XML tags by the ISY, and the TCP  and IP headers are added.  So filtering is absolutely necessary.

 

The answer is simple, don't wrap everything in XML tags, when a simple, efficient binary protocol will do. Especially for mobile push.

[jtara92101]

Here's what I would envision as the ideal scenario (James feel free to disagree with me/correct me):

 

1. ISY sends updates to ISY Portal for each state change (devices, scenes, variables, programs to start).

 

2. ISY Portal has an API for mobile app developers to hook into to subscribe to events from a particular ISY for their user. This can either be a continuous subscription socket, or connection initiated from ISY Portal Server on each event (ie REST call). Each mobile app vendor would run their own server that would subscribe to events from the ISY Portal.

 

3. Mobile App Vendor server submits the updates/notifications to the appropriate cloud messaging service (GCM/FCM/APNS) depending on what platform the user is running their app on.

 

4. Cloud messaging server pushes updates/notifications to app running on user's mobile device.

 

 

 

For those concerned about the volume of data transferred, if an efficient binary protocol is used (ie. not XML), 100 bytes per event would probably be more than generous. Assuming a very high 10000 events per day, would would be 1MB per day, or about 30 MB per month. Unless your ISY is connected to a 3G device in Iran without a roaming plan, that shouldn't be a concern for most people.

 

On the mobile device side, I believe Agave currently maintains an open socket with a subscription, so it's already receiving all the updates from an ISY, but with a much less efficient protocol, and as such is sucking up much more data (and no one is complaining about how much data Agave is using). What people should be complaining about, is that running a background process and keeping a socket open is sucking up BATTERY. Moving updates to being push-based rather than keeping a socket open will give longer battery life on your mobile device, and use less data than Agave is currently using. On mobile, battery life is everything.

 

Also, running background processes and keeping an open socket is not even really an option on IOS devices. So UDI implementing an API that mobile app developers solves the problems for all mobile app developers, regardless of whether they're developing on IOS, Android, Windows Mobile, Amazon Kindle Fire, Windows Desktop, Mac Desktop, etc.

 

What James wants will make for a better user experience, may stimulate competition for better mobile apps, and in general make cranky people like me happy

 

Exactly!

 

Again unsure of Android, but at least on iOS, apps do NOT have to stay running to receive push alerts. The push alert opens the app, which need not show UI when it receives the alert. (App decides what to do.) Assume Android is similar.

 

This also leave it up to each mobile software vendor to implement the actual mobile-platform push alerts. And, so, it is the mobile software vendor (not UDI) who will get in trouble if they abuse the intended use of the push service.

 

(Yesterday's brew-ha-ha with the Dash documentation viewer - Apple revoked developer status of the developer alleging some sort of ratings abuse)  makes me glad that I have no published apps in the app stores - at least not under my own or company name. I develop apps for internal use by multi-national corporations. I cannot afford to jeopardize my developer account by having something stupid happen like a perceived abuse of app store...)

Edited October 8, 2016 by jtara92101

[giesen]

Anatomy of an example IP packet for this application:

 

[iP Header - 20 bytes | TCP Header - 20 bytes | ISY ID - 6 Bytes | Device/Node/Scene ID - 6 Bytes | Subnode/Event ID - 6 Bytes | Value - 8 Bytes]

 

By my count, that's 66 bytes, and supports pushing 64 bit values  Even if you needed a few more fields for whatever reason, it's still a trivial amount of data.

[G W]

Bandwidth, and in particular limits, include packet overhead - so the few bytes blow up to a lot more by the time the event is wrapped in the XML tags by the ISY, and the TCP and IP headers are added. So filtering is absolutely necessary.

First of all, get rid of XML. Go to binary between the ISY and the Portal. UDI controls both ends so UDI can solve all of this.

 

AND if bandwidth is an issue DON'T use the portal.

 

S.A.T.T.P.

Best regards,

Gary Funk

[jtara92101]

Not so fast the the defenestration of the XML.

 

XML compresses quite nicely. And the connection from mobile vendor backend to UDI backend should be compressed.

 

As a developer, I prefer using JSON, as it is much easier to consume. But it's just as bloated as XML, and also as easily-compressed.

 

XML is more open to compatibly-modifying data structures in the future (and with good design practices) without forcing the need to update existing software.

 

JSON is more limited in the data structures it can represent, but works for most needs.

 

I wouldn't recommend a pure binary connection, as it would be very difficult to deal with schema changes over time.

 

There is a spec for "binary JSON". 

 

    http://bsonspec.org/

 

I do this for a living - write mobile apps that talk to APIs. I've worked with a couple of the U.S. National Laboratories on apps that front-end services that analyze energy efficiency of buildings. some quite complex data. One lab choose XML, another choose JSON. The XML one took a mess of code. The JSON one was easy-peasy. The work of "Lab X" has been taken over by "Lab Y", FWIW. The JSON one.

 

Currently, I write mobile software for dispatching service technicians. JSON APIs. Easy-peasy.

 

Either way - XML, JSON, or something else... KISS.

 

It is common to offer EITHER XML or JSON, as well. But of course makes the web service code more complicated.

 

While *I* prefer JSON, UDI no doubt has their reasons for XML. They've adopted it extensively, it fits with the way they've been doing things.

 

Either way, efficiency really isn't much of an issue, due to their compressibility, and has to be balanced with upgradability.

[G W]

Since it's the data between the UDI ISY and the UDI Portal, UDI can do what's best to transfer the data.

 

S.A.T.T.P.

Best regards,

Gary Funk

[mwester]

Well, ok, I can see that data is free and bandwidth is unlimited, at least according to the posters on this thread.  According to the logic above, I'd never bother with a dripping tap, because we'd calculate the number of drips per hour, the volume of each drip, calculate the gallons per month, decide that based on the personal experience of the other posters on the forum that anyone concerned about the bill for that amount of water has their water service delivered from Iran, and therefore is an idiot and should be dismissed as such.

 

Fine, so bandwidth is unlimited.  And data is free.  Let's talk about security.

 

Ah - no let's not.  Most times we discuss security here, somebody p***es and moans that they don't have anything to protect and the vulnerability is of such low risk that the person bringing it up must have their security delivered via drone from Iran, and is therefore an idiot and should be dismissed as such.

 

But regardless of forum opinion, I DO care about drips in my faucet.  And I DO care about useless, unnecessary uploads of data -- both from a cost standpoint, and as I consider my discomfort with this concept further, I've become concerned about the security aspects of uploading every event on my ISY up to the portal.  Doing that, frankly, makes the data the Google collects with Nest inconsequential!

 

But, hey - I get it.  Few here take security of IoT seriously.

 

So, a big Thank You to Gary Funk for the obvious solution -- since I'm the only one who cares, I should just cancel my portal subscription.  When this misfeature gets implemented, sans filtering, I shall do exactly that.

[giesen]

Not so fast the the defenestration of the XML.

 

XML compresses quite nicely. And the connection from mobile vendor backend to UDI backend should be compressed.

 

As a developer, I prefer using JSON, as it is much easier to consume. But it's just as bloated as XML, and also as easily-compressed.

 

XML is more open to compatibly-modifying data structures in the future (and with good design practices) without forcing the need to update existing software.

 

JSON is more limited in the data structures it can represent, but works for most needs.

 

I wouldn't recommend a pure binary connection, as it would be very difficult to deal with schema changes over time.

 

There is a spec for "binary JSON". 

 

    http://bsonspec.org/

 

I do this for a living - write mobile apps that talk to APIs. I've worked with a couple of the U.S. National Laboratories on apps that front-end services that analyze energy efficiency of buildings. some quite complex data. One lab choose XML, another choose JSON. The XML one took a mess of code. The JSON one was easy-peasy. The work of "Lab X" has been taken over by "Lab Y", FWIW. The JSON one.

 

Currently, I write mobile software for dispatching service technicians. JSON APIs. Easy-peasy.

 

Either way - XML, JSON, or something else... KISS.

 

It is common to offer EITHER XML or JSON, as well. But of course makes the web service code more complicated.

 

While *I* prefer JSON, UDI no doubt has their reasons for XML. They've adopted it extensively, it fits with the way they've been doing things.

 

Either way, efficiency really isn't much of an issue, due to their compressibility, and has to be balanced with upgradability.

 

 

I wholeheartedly agree that binary protocols are more difficult to work with, but SOAP swung the pendulum so entirely in the other direction. XML made the protocol so gratuitous that it's difficult to parse, uncompressed it takes up huge amounts of storage, and loses the advantage of ASCII protocols being human-readable. There's a reason no one writes any new implementations with SOAP/XML anymore. I have no issues with JSON and REST, in fact it's quite lovely to work with. In fact, I believe UDI intends to abandon SOAP/XML long term, and move entirely to REST.

 

My point about binary protocols was more to show that subscriptions need not consume inordinate amounts of data. Of course JSON can be efficient, and likely more than efficient enough for this use case. However, I believe even one of GCM's modes uses a binary protocol.

 

In any case, I don't really care what gets used, as long as some workable solution is implemented.

Edited October 8, 2016 by giesen

[giesen]

Well, ok, I can see that data is free and bandwidth is unlimited, at least according to the posters on this thread.  According to the logic above, I'd never bother with a dripping tap, because we'd calculate the number of drips per hour, the volume of each drip, calculate the gallons per month, decide that based on the personal experience of the other posters on the forum that anyone concerned about the bill for that amount of water has their water service delivered from Iran, and therefore is an idiot and should be dismissed as such.

 

Fine, so bandwidth is unlimited.  And data is free.  Let's talk about security.

 

Ah - no let's not.  Most times we discuss security here, somebody p***es and moans that they don't have anything to protect and the vulnerability is of such low risk that the person bringing it up must have their security delivered via drone from Iran, and is therefore an idiot and should be dismissed as such.

 

But regardless of forum opinion, I DO care about drips in my faucet.  And I DO care about useless, unnecessary uploads of data -- both from a cost standpoint, and as I consider my discomfort with this concept further, I've become concerned about the security aspects of uploading every event on my ISY up to the portal.  Doing that, frankly, makes the data the Google collects with Nest inconsequential!

 

But, hey - I get it.  Few here take security of IoT seriously.

 

So, a big Thank You to Gary Funk for the obvious solution -- since I'm the only one who cares, I should just cancel my portal subscription.  When this misfeature gets implemented, sans filtering, I shall do exactly that.

 

Data/bandwidth is relatively cheap. On the scales we're talking about, it's a rounding error on your cell phone data report. If you're concerned about your cell phone data usage, turn off push notifications for the app.

 

On the other end (the ISY side), on the average home internet plan (let's say your cap is 100GB, which is pretty low these days), it's %0.03 of your cap. Even for those with cottages in the sticks where you might only get get 5GB, it's still only %0.6 of your cap. So let's keep the scale of this (non-)problem in perspective.

 

No one is talking about having every level 3 debug log sent automatically to the portal (or at least I'm not). What we're talking about is sending device events; a light switch turned on, a motion sensor tripped, the thermostat calling for heat. That type of thing. Yes, it needs to be appropriately protected, but the reality is if the UDI portal is hacked (or your portal account is hijacked), everything on your ISY is available to the hacker anyways. They just have to poll the data from your ISY, instead of it being automatically uploaded. Push notifications changes nothing about that. The state information doesn't even have to be stored on the portal once the notifications are forwarded. What it does do is make it more efficient and provides a better user experience for mobile users.

Edited October 8, 2016 by giesen

[mwester]

No one is talking about having every level 3 debug log sent automatically to the portal (or at least I'm not). What we're talking about is sending device events; a light switch turned on, a motion sensor tripped, the thermostat calling for heat. That type of thing. Yes, it needs to be appropriately protected, but the reality is if the UDI portal is hacked (or your portal account is hijacked), everything on your ISY is available to the hacker anyways. They just have to poll the data from your ISY, instead of it being automatically uploaded. Push notifications changes nothing about that. What it does do is make it more efficient and provides a better user experience for mobile users.

 

I disagree.

 

Reason 1: Fundamental security concept: Defense-in-depth.  By analogy, you argue that there's no difference in putting your wallet, jewelry, etc. on the counter in your mudroom relative to putting that stuff away in a drawer in your bedroom (if not in a safe in your house).  After all, you argue, if someone is going to kick in your door, they can just as easily go to your bedroom and find your jewelry as they can snatch it off the counter by the door and run.   Clearly that's not true -- there IS value in making that stuff harder to get.

 

Reason 2: Breaking into the portal and polling my device for data is quite different -- first of all, it requires a lot more knowledge and skill than just stealing a database or file with thousands of events recorded.  Secondly, I can detect that activity at my firewall, if someone is polling my device -- I can't detect that someone has stolen that information at the portal, nor can UDI detect that.  So clearly, security principles suggest that the former is the correct way to handle things.

 

This very discussion illustrates what I've observed.  People will rationalize all sorts of compromises on basic security concepts, just to gain convenience.  Nowhere in this thread have I threatened ANYONE's convenience -- I've asked ONLY that it be filtered so that those of us who DO care don't have to drop the portal altogether.

[MWareman]

Anatomy of an example IP packet for this application:

 

[iP Header - 20 bytes | TCP Header - 20 bytes | ISY ID - 6 Bytes | Device/Node/Scene ID - 6 Bytes | Subnode/Event ID - 6 Bytes | Value - 8 Bytes]

 

By my count, that's 66 bytes, and supports pushing 64 bit values Even if you needed a few more fields for whatever reason, it's still a trivial amount of data.

I think you are missing encryption overhead.....

[giesen]

I disagree.

 

Reason 1: Fundamental security concept: Defense-in-depth.  By analogy, you argue that there's no difference in putting your wallet, jewelry, etc. on the counter in your mudroom relative to putting that stuff away in a drawer in your bedroom (if not in a safe in your house).  After all, you argue, if someone is going to kick in your door, they can just as easily go to your bedroom and find your jewelry as they can snatch it off the counter by the door and run.   Clearly that's not true -- there IS value in making that stuff harder to get.

 

Reason 2: Breaking into the portal and polling my device for data is quite different -- first of all, it requires a lot more knowledge and skill than just stealing a database or file with thousands of events recorded.  Secondly, I can detect that activity at my firewall, if someone is polling my device -- I can't detect that someone has stolen that information at the portal, nor can UDI detect that.  So clearly, security principles suggest that the former is the correct way to handle things.

 

This very discussion illustrates what I've observed.  People will rationalize all sorts of compromises on basic security concepts, just to gain convenience.  Nowhere in this thread have I threatened ANYONE's convenience -- I've asked ONLY that it be filtered so that those of us who DO care don't have to drop the portal altogether.

 

 

And you didn't actually read what I said. The events are not stored in persistent storage in the portal. Just long enough to send the event. You think Google or Apple store all the push notifications that are sent to them?

 

And the crux of your original argument for filtering was bandwidth usage. Which has clearly been debunked. You had not even brought up a security argument until 30 minutes ago. As to your argument about seeing the traffic on your firewall, unless you're doing SSL interception on your firewall, all you're likely to see are a bunch of packets with no indication of their contents save for the fact they're going to your ISY. Hardly something conclusive to unleash the hounds upon.

 

Don't get me wrong, I agree security is important. I'm also not opposed to any sort of event filtering. I don't think things like the logs need to be (or should be) sent to the portal. They can be pulled on request. And if you want to filter out events for certain devices/scenes/etc, I'd support that. But let's keep arguments rational and consistent.

Edited October 8, 2016 by giesen

[giesen]

I think you are missing encryption overhead.....

 

It was an example to demonstrate the scale of the (non-)problem. Even if the packet was 4-5 times as large, it would still be a trivial amount of data.

[G W]

Well, ok, I can see that data is free and bandwidth is unlimited, at least according to the posters on this thread. According to the logic above, I'd never bother with a dripping tap, because we'd calculate the number of drips per hour, the volume of each drip, calculate the gallons per month, decide that based on the personal experience of the other posters on the forum that anyone concerned about the bill for that amount of water has their water service delivered from Iran, and therefore is an idiot and should be dismissed as such.

 

Fine, so bandwidth is unlimited. And data is free. Let's talk about security.

 

Ah - no let's not. Most times we discuss security here, somebody p***es and moans that they don't have anything to protect and the vulnerability is of such low risk that the person bringing it up must have their security delivered via drone from Iran, and is therefore an idiot and should be dismissed as such.

 

But regardless of forum opinion, I DO care about drips in my faucet. And I DO care about useless, unnecessary uploads of data -- both from a cost standpoint, and as I consider my discomfort with this concept further, I've become concerned about the security aspects of uploading every event on my ISY up to the portal. Doing that, frankly, makes the data the Google collects with Nest inconsequential!

 

But, hey - I get it. Few here take security of IoT seriously.

 

So, a big Thank You to Gary Funk for the obvious solution -- since I'm the only one who cares, I should just cancel my portal subscription. When this misfeature gets implemented, sans filtering, I shall do exactly that.

If you have concerns DON'T use it. No one is going to force you to use the feature.

 

S.A.T.T.P.

Best regards,

Gary Funk

[mwester]

If you have concerns DON'T use it. No one is going to force you to use the feature.

 

S.A.T.T.P.

Best regards,

Gary Funk

 

That's exactly what I'm asking for - a filter, so I don't have to use this feature!!!

[mwester]

And you didn't actually read what I said. The events are not stored in persistent storage in the portal. Just long enough to send the event. You think Google or Apple store all the push notifications that are sent to them?

 

And the crux of your original argument for filtering was bandwidth usage. Which has clearly been debunked. You had not even brought up a security argument until 30 minutes ago. As to your argument about seeing the traffic on your firewall, unless you're doing SSL interception on your firewall, all you're likely to see are a bunch of packets with no indication of their contents save for the fact they're going to your ISY. Hardly something conclusive to unleash the hounds upon.

 

Don't get me wrong, I agree security is important. I'm also not opposed to any sort of event filtering. I don't think things like the logs need to be (or should be) sent to the portal. They can be pulled on request. And if you want to filter out events for certain devices/scenes/etc, I'd support that. But let's keep arguments rational and consistent.

 

 

Yes, events WILL be stored.  For seconds, or milliseconds -- but the data WILL be stored for at least some point in time.  And during that time, it's not under my control.

 

Now, as for the second paragraph there -- I apologize.  I sincerely did not understand that one is limited to one argument per topic.   I retract all my concerns about security, and I guess I'll start another thread for that argument on this.

 

Finally, I do apologize for offending you with my irrational and my inconsistent arguments.  My training, if its any excuse, is in the area of software and systems security, and my experience arguing a case in front of a judge is pretty much zero.

 

Sigh.  Off you go then, enjoy an argument-free, conflict-free discussion, one that follows your rules as you make them up.

 

[The above is, of course, sarcasm.  This is not a legal court where arguments are presented to a judge - this is a forum, where ideas are discussed, explored, debated, and argued.  There are no rules that state one is limited to one position or argument per topic.  Nor are there rules limiting new ideas and thoughts.  However, one IS free to make up their own rules as they wish -- but alas, nobody else is obligated in any way to follow those made-up rules.

 

This discussion is an exploration of the issues involved in streaming events up to a cloud-based service; there are those who do not want to hear about any issues, and those who wish to marginalize those who bring up the issues in order to avoid discussing the technical merits.  I hope that those who would implement can acknowledge the concerns of those who've brought up the issues, and implement the very simple mitigation that's been suggested.]

[Teken]

I truly believe some of you just need to unplug from this conversation.

 

Let's all keep this real it doesn't matter what anyone thinks it's up to UDI to move forward with this idea. Assuming if it's even doable in terms of ROI.

 

Nothing saddens me more than to see long term members argue on points that are valid on both sides.

☹️️

 

Take the good, bad, ugly, and throw away the rest.

 

Good grief - Some of you just like to argue and get others into a tizzy!

 

That's my job!

 

 

=========================

 

The highest calling in life is to serve ones country faithfully - Teach others what can be. Do what is right and not what is popular.

[G W]

I truly believe some of you just need to unplug from this conversation.

 

Let's all keep this real it doesn't matter what anyone thinks it's up to UDI to move forward with this idea. Assuming if it's even doable in terms of ROI.

 

Nothing saddens me more than to see long term members argue on points that are valid on both sides.

☹️️

 

Take the good, bad, ugly, and throw away the rest.

 

Good grief - Some of you just like to argue and get others into a tizzy!

 

That's my job!

 

 

=========================

 

The highest calling in life is to serve ones country faithfully - Teach others what can be. Do what is right and not what is popular.

Yes it is.

 

Best regards,

Gary Funk

[Michel Kohanim]

Hello everyone,

 

Thanks for all the feedback and heated discussions. From UDI perspective:

1. Having a filter on ISY Portal is not a good idea because a) we will need more subscription servers inefficient use of bandwidth for things that are meaningless (such as those send on Event Viewer Level 3 - debugging information) and c) events will be portal dependent

2. Changing the packets to use binary is not a good idea since it does not follow any standards (either internally or externally) and it's not backward compatible. We will then have to have glue code that decides to interpret things based on the URL. Either using XML or transforming to JSON is a much better approach

 

We do know that we can use programs do send notifications (including push). The question is if we can make this process (having programs that send push notification) easier with some APIs, would that suffice?

 

With kind regards,

Michel