ISY Portal Says devices not available.

[palayman]

4 minutes ago, hum099 said:

Mine is working now

Me too!

[Condor]

Not working here. ISY portal appears to be offline and Alexa integration is not working. Attempting to access the ISY portal results in "HTTP error 500". Local control is working fine. I haven't tried re-booting yet since it doesn't seem to have worked for most of the folks reporting problems right now. 

[bmercier]

This should be fixed for everyone now. Currently doing additional checks.

Will post a summary later.

[Condor]

It's working for me now. The ISY portal is online and Alexa integration is responding. I did not need to reboot. 

[bmercier]

So here's what happened.

There were 2 bugs in the code of the proxy servers that were dormant. They had no effects, unless a proxy server had a hard stop... which happened last night. 

The first bug had the effect that some units were trying to connect to a no longer existing proxy server.

The second bug is more complicated to explain, but it was also caused by the hard stop. This one was causing the HTTP 500.

The hard stop was caused by AWS stopping one of our proxy instance unexpectedly. This was NOT a graceful shutdown. This is usually caused by the failure of the underlying hardware.

Both bugs have been fixed and are in production.

[dantoronto]

thanks so much for all your skill and patience.  I agree with the comment that we rarely see outages.  But clearly many of us are highly dependent on this service.  Can't imagine a life where I actually have to touch a light switch!!