Portal dead again?

[RMcQuiston]

Suddenly, my lights are responsive to Alexa devices.  Tried relinking but now I get "We were unable to link ISY Optimized for Smart Home V3 at this time."

 

Anyone else??

[bmercier]

It's up now. ISYs are reconnecting.

[Illusion]

Still offline for me

Edit: There she goes! Back online.

 

Edited December 3, 2022 by Illusion

[bmercier]

Just now, Illusion said:

Still offline for me

 

May take a few minutes to have all ISYs to reconnect.

[jwagner010]

I am back. Hooray.  Thank you. 

[sdcrane]

Back up for me.  Thanks!

[lsibarra]

It looks like the Portal is back-up.

 

Thank you @bmercierfor fixing the portal.

 

Larry

[IndyUDIuser]

My Alexa and phone app weren't working but they're back up now.  Thanks!

[bmercier]

I know it's been 3 outages in just a few days, and this looks really bad on us. 

Let me share with you a few more details.

The first outage was due to a stupid lack of disk space on the database server. Logs suddenly started to grow quickly until we ran out of space. Clearing the logs allowed us to get back online quickly.

Second outage was more subtle and was due to a performance problem with the oAuth server. It went slower and slower until it reached a threshold at which point Alexa was making retries on the oAuth server, which quickly worsened the problem with deadlocks on oAuth records, which ultimately made the database server reach 100% cpu in an instant, which made everything terribly slow. A simple index added fixed the problem.

This time again is a different problem. It appears to be an Amazon failure where connection to the database server was severed. I could not even SSH to the server, all communication was stopped. I experienced this type of problems a few times over the years, and due to redundancy it never had an impact on the uptime. But when it happens on the database server like this time, it's a different story.

So this is really 3 different and unrelated problem which happened in a short period of time.

[Condor]

Thank you for the detailed outage analysis. Knowing what is happening "behind the scenes" is very helpful. Thanks again!

PS. The portal is working again for me too.

[DennisC]

Thanks for taking care of the issue so quickly and for the explanation of what was happening.

Very much appreciated!

[Yoda]

Thanks.

Have you taken any steps to increase available disk space for the long term?

[dbwarner5]

Lesson learned for me on this thread:

- when Alexa is down, check the portal via the AC.

 

Luckily for me, when I noticed Alexa wasn't working, I had just finished installing my new matter board and doing backups etc, and it time for a nap, so I didnt waste any time trying to "fix" the Alexa problem.

As stated before, UD is generally so stable, I would not have thought of the portal immediately.

Thanks @bmercier. while I sleep, the elves were busy.......

Edited December 3, 2022 by dbwarner5
typos

[kcrimson]

Back up for me also!! Thanks for quick response. 

[bmercier]

2 hours ago, Yoda said:

Thanks.

Have you taken any steps to increase available disk space for the long term?

No, there is plenty of disk space already