websockets from non-ISY server

[MarkJames]

I'm an idiot.  My son (Jordan) had put some stuff in there and I took it out for debugging.  I took out the js folder by mistake

 

Your last changes....

 

.

.

.

.

.

.

.

.

 

WORKED!!!!!!

 

Thank you ever so much!

 

I'm going to compile the process into a separate thread for those who may follow.

 

Thanks, Michael - I would NEVER have gotten this working without your help!

 

mark

[MWareman]

Great to hear!!  Now - don't forget to change the password for 'mark' in /etc/htpasswd-isy

(or even use a different username!)

 

I've updated the wiki page with our findings on the needed modules...  http://wiki.universal-devices.com/index.php?title=Apache_Reverse_Proxy

 

I'm glad we got it all figured out! (BTW - a great looking site you're putting together!)

 

Michael.

[MarkJames]

Thanks!

 

As you're putting it in the wiki I won't rewrite what you're already doing.  I'll add the following comments, though.

 

On the Raspberry Pi 3 the virtual hosts file is located at /etc/apache2/sites-available/000-default.conf - be sure to back this up before changing it.  

There are a few modules that need to be enabled 

 

#sudo a2enmod authz_groupfile

#sudo a2enmod proxy_http

#sudo a2enmod proxy_html

sudo a2enmod headers

 

 

Better go hang out with my wife for a bit - she's giving me the evil eye.

 

Thanks so much!  I'm excited - I just have to find some documentation on the xml that's getting passed from the subscription now.

 

mark

 

 

Don't forget the a2enmod on the authz_groupfile and the 

[MWareman]

Thanks!

 

As you're putting it in the wiki I won't rewrite what you're already doing.  I'll add the following comments, though.

 

On the Raspberry Pi 3 the virtual hosts file is located at /etc/apache2/sites-available/000-default.conf - be sure to back this up before changing it.  

There are a few modules that need to be enabled 

 

#sudo a2enmod authz_groupfile

#sudo a2enmod proxy_http

#sudo a2enmod proxy_html

sudo a2enmod headers

 

 

Better go hang out with my wife for a bit - she's giving me the evil eye.

 

Thanks so much!  I'm excited - I just have to find some documentation on the xml that's getting passed from the subscription now.

 

mark

 

 

Don't forget the a2enmod on the authz_groupfile and the 

 

According to http://wiki.universal-devices.com/index.php?title=ISY_Developers:API:REST_Interface#Subscriptions_.28Web_Sockets.29 - it's section 5 in ISY WS-SDK Manual.

 

The authz_groupfile is not needed now - I've removed the line that needed it (it wasn't doing anything anyway). 

 

Off to grill now...  

[MarkJames]

Great stuff.  I already have my socket connected and the listener is  updating my device statuses in real time.  This is very cool.  I'm just going to go through the wiki to find what all the actions, eventinfo, nodes, etc. are in the string that comes from the subscription.

 

Oh - you may want to add a line about the userid:password needing to be base64 encoded in the wiki entry.  There are numerous online sites to do the encoding.

 

Once again - thanks Michael - your expertise and willingness to share it is very much appreciated.

 

mark

[MWareman]

Thanks for your kind comments Mark...

 

I've added details into the wiki. You can actually (probably) generate the needed string directly on your RPI ("echo -n "user:password" | base64") - so you don't have to send your password off to who-knows where...

 

To be honest - I hope that the HAD code (and UDAjax) can be enhanced with a subscription and real time updates. This is exactly the use case websockets are for!

 

One thing (don't know if you've noticed this yet) - on first subscribing you get a dump of all devices current status. There is no need to separately get status as you are building the page. It would be useful for keeping the code there for a manual update (should it be necessary), but this might help make the load time even better.

 

Michael.

[MarkJames]

Sounds good generating the base64 in-house.  All I had in mind was that it be mentioned in the wiki what information was requested in the authentication directive - it's not intuitively obvious (at least it wasn't for me).

 

I did notice that there was a data dump at first - there was also an undefined element that was problematic at first - I dealt with that with just a try{}.  At the moment I don't update my devices singly - I update every one of the devices, programs, and variables that I track.  Ultimately I'll probably change it but so far it all happens so quickly that It's on the back burner.  Because of this I had to ignore the initial dump.  Here's the code I've been using so far.

 

var parseDisabled = true; // don't parse for the first 2 seconds

$(document).ready(function() {
setTimeout(function() {
        parseDisabled = false;
        }, 2000);
 var urlvalue;
 var socket;

 var isy_host = window.location.host;
 var x = location.protocol;
 if (x == "https:") {
   var ws_host = "wss://" + isy_host
 } else {
    var ws_host = "ws://" + isy_host
  }
 urlvalue = ws_host + "/rest/subscribe";
 socket = new WebSocket(urlvalue, "ISYSUB");
 socket.addEventListener("message", function(event) {
 isyParse(event.data);
        });
.
.
. rest of the document.ready function

// then this is the beginning of my parse routine to determine what kind of message it is and how to deal with it. There are a lot of messages and I'm only just beginning
// so far it just looks for "ST" and triggers an update of all the elements on my page as a proof of concept.
// the parseDisabled variable is just a 2 second delay from initial page load so I don't do an updateStatusAll() on every single one of the items
// in the initial dump
// Once the update routine does individual elements I'll be able to ditch that
function isyParse(data) {
    if (parseDisabled) {
        return;
    }
    var parser = new DOMParser();
    data = parser.parseFromString(data, "text/xml");
    try {
    var elementID = data.getElementsByTagName("control")[0];
    var nodeID = data.getElementsByTagName("node")[0];
    var status = data.getElementsByTagName("action")[0];
   
    if (elementID.childNodes[0].nodeValue=="ST") {
        var node=nodeID.childNodes[0].nodeValue;
        var action =  status.childNodes[0].nodeValue;
         updateStatusAll();// this is the routine that updates all my colors, icons, and text
        console.log(node+" turned "+(action == 0 ? "off" : "on"));
    }
    }catch(exc) {  }
}

[Michel Kohanim]

Hi MWareman,

 

Just wanted to THANK YOU. This is fantastic.

 

With kind regards,

Michel

[MWareman]

All,

 

I have added a couple of things to the example on the wiki.

 

The first is a line in the '<Proxy>' definition that I *highly* recommend you add. With Authentication, this are secure - but if anyone breaks the authentication (or you misconfigure it) Apache would become an open proxy. You don't want that!  Please add a 'Require host lights.domain.com' (modify for your own DNS name) so that requests are limited to your specific host. That way - things are kept more secure.

 

Second, I have rearranged the authentication rules....  to allow another feature.....

 

You can (manually) generate a secure nonce, and have Apache proxy requests to the rest API using the nonce without needing auth at all. This is very useful for receiving incoming requests from, for example, the IFTTT Maker channel. If requires SSL though - so is an advanced use case. The three lines implementing this are commented out in the example by default.

 

Michael.

[MarkJames]

Thanks for your ongoing embellishment on this, Mike.

 

Can you elaborate a bit on this for me?  And if this is OT then we can do this in PM instead.  Having gone through my life with 'security through obscurity' for these many years I'm not sure what to do here.

 

1. What would need to change so far as my Apache webserver if I choose to go with SSL?  I assume I'd need to change the port from 80 to 443.
2. would there need to be changes to httpd.conf?  Would it be possible to have both a secure and non-secure route to the website?
3. Would I need to buy a certificate and if so would it be a standard certificate or a wildcard one?  Bearing in mind that my site is a single page and wouldn't lend itself to separating out into a secure/non secure area.  The cost at Garrison of a standard certificate was $15 a year - $45 for wildcard.  It's not a big expenditure but it bears thinking about.
4. What would be the benefit over, say, a PHP or .htaccess login.  Other than the plaintext transmission of the password which, though serious for e-commerce doesn't really concern me much for my own uses.

Sorry for the noob questions but .... I'm a noob

 

makr

[MWareman]

There would be a port change and other changes to the httpd.conf. when I get home later I'll post the needed changes.

 

You do also need a trusted SSL certificate - especially if you want to be able to receive IFTTT requests. I have a wildcard cert from these guys (http://www.garrisonhost.com/ssl-certificates/alphassl.html), but if your is a single name site then the regular cert will work just fine. At $15/yr it's the cheapest trusted cert I've found.

 

I use a wildcard because I host several sites each with their own name all sharing the same certificate (for my access point management software, firewall, log correlation etc...).

 

The .conf I use has a very simple host on port 80 that redirects to https - so a user can do either, but always end up on the secure site.

 

I did notice during testing you have your alarm system integrated. There is more than enough info to locate you, so I'd spend the $15/yr to secure the login.

 

Michael.

[MarkJames]

 

I did notice during testing you have your alarm system integrated. There is more than enough info to locate you, so I'd spend the $15/yr to secure the login.

 

 

 

I'm inclined to go SSL if for nothing but the 'geek' quotient.

 

You have to bear in mind that I *did* give you the password to get past .htaccess, though.  At the moment each of my family members has one but the general public doesn't.  While a cleartext password like .htaccess uses is easily susceptible to a man-in-the-middle or other attack it's probably plenty secure for my purposes.  I could also go PHP with an encrypted password stored outside the web structure to start a session and be reasonably secure....

 

I'll wait and see what you post in this regard.  I'll likely go along - at least for a year - to see how it works.  IFTTT looks like it could be cool, too.

 

mark

[MWareman]

A php based system is still not secure without a certificate - as the authentication transaction is still in the clear. Only SSL can close this gap.

 

For the client to authenticate to the proxy, Apache must be able to handle it before passing control to .php. that's why auth is in the .conf here. It's also important that the same username/password and resulting session is used for both the web interface and calls to the /rest interface, and the websocket connection. I doubt the php based auth will allow websockets to work at all.

 

Michael.

[MarkJames]

You're the guy who understands this stuff!

 

I'm the guy who follows along and hopes things will work out.

 

I'll do my job - you do yours

 

 

I'll wait to see what you come up with and be the 'patient zero' (or near zero) tester.  

 

Thanks!

 

mark

[MWareman]

Here is my full config file (with certain values changed...) with the redirect from port 80 and SSL config.

 

I got my cert from here: http://www.garrisonhost.com/ssl-certificates/alphassl.html

The standard cert is fine. You don't have to use this CA though - that's up to you. Make sure the SSLCertificateFile points to the certificate, SSLCertificateKeyFile the private key and SSLCertificatechainFile to the intermediate certificate provided by the CA.

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin webmaster@lights.domain.com
        ServerName lights.domain.com
        DocumentRoot /var/www/lights
        ProxyRequests Off
        ProxyPreserveHost On
        KeepAlive On
        KeepAliveTimeout 5000
        ProxyVia Off
        #SetEnvIf Request_URI ^/{SECRET_KEY}/ noauth=1
        <Proxy *>
                Require host lights.domain.com
                AuthName "Authentication Required"
                AuthType Basic
                AuthUserFile /etc/htpasswd-isy
                AuthGroupFile /dev/null
                Order deny,allow
                Satisfy any
                Deny from all
                require valid-user
                Allow from env=noauth
        </Proxy>
        RequestHeader set Authorization "Basic {ISY_BASE64_CREDS}"
        ProxyPass "/rest/subscribe" "ws://192.168.1.2/rest/subscribe" retry=4
        ProxyPass /rest http://192.168.1.2/rest
        ProxyPass /services http://192.168.1.2/services
        ProxyPass /WEB http://192.168.1.2/WEB
        ProxyPass /USER http://192.168.1.2/USER
        #ProxyPass /{SECRET_KEY} http://192.168.1.2/rest
        #ProxyPassReverse /{SECRET_KEY} http://192.168.1.2/rest
        ProxyPassReverse "/rest/subscribe" "ws://192.168.1.2/rest/subscribe" retry=4
        ProxyPassReverse /rest http://192.168.1.2/rest
        ProxyPassReverse /services http://192.168.1.2/services
        ProxyPassReverse /WEB http://192.168.1.2/WEB
        ProxyPassReverse /USER http://192.168.1.2/USER
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ErrorLog ${APACHE_LOG_DIR}/error.log
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/public.pem
        SSLCertificateKeyFile /etc/ssl/private/private.key
        SSLCertificateChainFile /etc/ssl/chain.crt
</VirtualHost>
</IfModule>

<VirtualHost *:80>
        ServerAdmin webmaster@lights.domain.com
        ServerName lights.domain.com
        RedirectMatch ^/(.*)$ https://lights.domain.com/$1
</VirtualHost>

If you want to be able to accept API calls from IFTTT, generate a secure, random value in some trusted way. Replace {SECRET_KEY} with this value on the three lines it appears, and un-comment the lines.

 

If the REST call was https://lights.domain.com/rest/status (needing basic auth) - then the API call https://lights.domain.com/{SECRET_KEY}/status will be successful without basic authentication. Obviously - keep {SECRET_KEY} secret! This relies on SSL and the cryptographic strength of {SECRET_KEY} for security. 

 

That being said - I'm now successfully having my Foscams trigger events on my ISY via this method... (they cannot do basic auth) - so this makes me very happy!

 

Michael.

[MarkJames]

If the REST call was https://lights.domain.com/rest/status (needing basic auth) - then the APIcall https://lights.domain.com/{SECRET_KEY}/status will be successful without basic authentication. Obviously - keep {SECRET_KEY} secret! This relies on SSL and the cryptographic strength of {SECRET_KEY} for security. 

 

 

Sorry if this is a dumb question but if I have to embed an api call that contains my secret key in my index.html file then how does it stay secret?

 

mark

[MWareman]

You don't. As you say - if you do that the secret does not stay secret!

Browser based api calls use the regular URLs and will send an authorization header with their requests - based on the primary authentication to the site.

The secret based access is for a service like IFTTT - so you can set it up without having to give them your actual password. This means that if they ever get compromised, your password is still safe and you don't have to reconfigure all your devices. All you have to do is change the key to re-secure it.

Michael.

[MarkJames]

Seems like I have a problem with the proxypass setup, Mike.  I noted that my /var/log/apache2/error.log grew to 23gb in the last few days.  I truncated it and started it over so that I could open it in nano.  

 

This is what it's full of

 

[Sat Jul 09 11:08:43.809026 2016] [proxy_http:error] [pid 652] (20014)Internal error: [client 192.168.0.1:49976] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeon$
[Sat Jul 09 11:08:44.022473 2016] [proxy_http:error] [pid 971] (20014)Internal error: [client 192.168.0.1:49979] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeon$
[Sat Jul 09 11:08:44.091199 2016] [proxy_http:error] [pid 651] (20014)Internal error: [client 192.168.0.1:49982] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeon$

Any thoughts on what this means?  192.168.0.1 is my router, 192.168.0.171 is my ISY.  Looks like the port forward to my webserver is causing the problem?

 

mark

 

[MWareman]

Mark, I'm not getting these errors on my setup. Also, port forwarded connections wouldn't change the source IP to the IP of your router.

 

This log is indicating regular connections from your router itself to some url on the Apache box, and its hitting a proxied url. It's obviously not getting a response it's expecting, and retrying at quite the rate!

 

What type of router do you use? Does it have any kind of intelligence, like device identification? If we can identify the url that the router is requesting we can exempt it from being proxied, and the errors will probably go away.

 

Michael.

[MarkJames]

I'm using an Asus N66-RTU with Tomato firmware.  

 

I had logging - including web logging to my readynas - enabled.  I've disabled it now.  I also had VPN tunneling and PPTP enabled and I've disabled those as well.

 

None of these should be hitting the Raspi on 192.168.0.233, though....

[MWareman]

That's very odd - I agree that none of that should be the source. There is something different though.

 

I don't use Tomato - I'll have to look at it capabilities to know if any of its configuration options could cause this. Does it have any kind of service monitoring?

 

I'm also interested by the http://homeon$ referer. It's not something I can find any info on - even in the Apache source code. It adds credibility to the idea that this isn't anything in the Apache proxy code itself causing this. Does it offer you any ideas?

 

Michael.

[MarkJames]

I'd assumed the homeon$ had something to do with my domain name being homeonthewater.com.  Some sort of abbreviation?

 

I'll keep checking on the log file and see if things have improved any.  It was pretty funny - my Raspi has a 32gb SD card in it and all that's on it is Raspbian, Apache, and support files plus my website.  Definitely less than a gb.  I went to copy the xml file to it and got an 'out of disk space' error.  Took a while to find where all the space went.

 

mark

[MarkJames]

Sorry - I didn't realize that nano was truncating the lines... the full line looks like this

 

2016] [proxy_http:error] [pid 5353] (20014)Internal error: [client 192.168.0.1:49619] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/index.html

[sat Jul 09 18:07:25.037891 2016] [proxy_http:error] [pid 5354] (20014)Internal error: [client 192.168.0.1:49622] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/index.html

[sat Jul 09 18:07:25.195597 2016] [proxy_http:error] [pid 5355] (20014)Internal error: [client 192.168.0.1:49625] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/index.html

[sat Jul 09 18:07:25.264558 2016] [proxy_http:error] [pid 5356] (20014)Internal error: [client 192.168.0.1:49627] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/

[sat Jul 09 18:07:25.295611 2016] [proxy_http:error] [pid 5357] (20014)Internal error: [client 192.168.0.1:49630] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/

[sat Jul 09 18:07:25.515923 2016] [proxy_http:error] [pid 5354] (20014)Internal error: [client 192.168.0.1:49637] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/

[sat Jul 09 18:07:25.532304 2016] [proxy_http:error] [pid 5353] (20014)Internal error: [client 192.168.0.1:49634] AH01102: error reading status line from remote server 192.168.0.171:80, referer: http://homeonthewater.com/

 

So it's my own webpage that's hammering the ISY and getting 'error reading status line'.

 

I have no idea how to troubleshoot this....

 

mark

[MarkJames]

Seems to be solved though I don't know why.

 

I added two lines to the 000-default.conf 

 

Do you know what these do and are they problematic in other ways?

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

[MWareman]

 

Seems to be solved though I don't know why.

 

I added two lines to the 000-default.conf 

 

Do you know what these do and are they problematic in other ways?

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

 

Basically, this downgrades the proxied requests to HTTP/1.0 from HTTP/1.1 - and in doing so disables HTTP keepalives... 

 

I feel this is a safe thing to do - and probably an advised thing to do. As far as I know, ISY does not fully support HTTP/1.1 or keepalives....

 

I've tested this - and it performs well. I've added this to the example in the wiki.

 

Thanks for finding this!

 

Michael.