RMcQuiston Posted December 3, 2022 Posted December 3, 2022 Suddenly, my lights are responsive to Alexa devices. Tried relinking but now I get "We were unable to link ISY Optimized for Smart Home V3 at this time." Anyone else??
Solution bmercier Posted December 3, 2022 Solution Posted December 3, 2022 It's up now. ISYs are reconnecting. 1 1
Illusion Posted December 3, 2022 Posted December 3, 2022 (edited) Still offline for me Edit: There she goes! Back online. Edited December 3, 2022 by Illusion
bmercier Posted December 3, 2022 Posted December 3, 2022 Just now, Illusion said: Still offline for me May take a few minutes to have all ISYs to reconnect. 1
lsibarra Posted December 3, 2022 Posted December 3, 2022 It looks like the Portal is back-up. Thank you @bmercierfor fixing the portal. Larry
IndyUDIuser Posted December 3, 2022 Posted December 3, 2022 My Alexa and phone app weren't working but they're back up now. Thanks!
bmercier Posted December 3, 2022 Posted December 3, 2022 I know it's been 3 outages in just a few days, and this looks really bad on us. Let me share with you a few more details. The first outage was due to a stupid lack of disk space on the database server. Logs suddenly started to grow quickly until we ran out of space. Clearing the logs allowed us to get back online quickly. Second outage was more subtle and was due to a performance problem with the oAuth server. It went slower and slower until it reached a threshold at which point Alexa was making retries on the oAuth server, which quickly worsened the problem with deadlocks on oAuth records, which ultimately made the database server reach 100% cpu in an instant, which made everything terribly slow. A simple index added fixed the problem. This time again is a different problem. It appears to be an Amazon failure where connection to the database server was severed. I could not even SSH to the server, all communication was stopped. I experienced this type of problems a few times over the years, and due to redundancy it never had an impact on the uptime. But when it happens on the database server like this time, it's a different story. So this is really 3 different and unrelated problem which happened in a short period of time. 5 6
Condor Posted December 3, 2022 Posted December 3, 2022 Thank you for the detailed outage analysis. Knowing what is happening "behind the scenes" is very helpful. Thanks again! PS. The portal is working again for me too.
DennisC Posted December 3, 2022 Posted December 3, 2022 Thanks for taking care of the issue so quickly and for the explanation of what was happening. Very much appreciated!
Yoda Posted December 3, 2022 Posted December 3, 2022 Thanks. Have you taken any steps to increase available disk space for the long term?
dbwarner5 Posted December 3, 2022 Posted December 3, 2022 (edited) Lesson learned for me on this thread: - when Alexa is down, check the portal via the AC. Luckily for me, when I noticed Alexa wasn't working, I had just finished installing my new matter board and doing backups etc, and it time for a nap, so I didnt waste any time trying to "fix" the Alexa problem. As stated before, UD is generally so stable, I would not have thought of the portal immediately. Thanks @bmercier. while I sleep, the elves were busy....... Edited December 3, 2022 by dbwarner5 typos 1 1
kcrimson Posted December 4, 2022 Posted December 4, 2022 Back up for me also!! Thanks for quick response.
bmercier Posted December 4, 2022 Posted December 4, 2022 2 hours ago, Yoda said: Thanks. Have you taken any steps to increase available disk space for the long term? No, there is plenty of disk space already
Recommended Posts