Tedsan Posted September 26, 2021 Posted September 26, 2021 Down in Pennsylvania. Comcast is my network provider
johnstonf Posted September 26, 2021 Posted September 26, 2021 My ISY was horribly slow, rebooted, now fast and online again. What happened, or is happening?
BMarking Posted September 26, 2021 Posted September 26, 2021 8 minutes ago, blakevh said: Alexa is happy again. The portal is online for me. Yep, didn't even have to buy her flowers.
romeo_echo Posted September 26, 2021 Posted September 26, 2021 online for me now since 11:07am Mountain time Sunday. Thanks for fixing!
Florie Posted September 26, 2021 Posted September 26, 2021 Portal integration is back online again, Thank you!
Tedsan Posted September 26, 2021 Posted September 26, 2021 1 hour ago, Tedsan said: Down in Pennsylvania. Comcast is my network provider Back on line in PA
johnstonf Posted September 26, 2021 Posted September 26, 2021 Has anyone figured out what hppened or is happening? This was a HUGE pain!
chevykid Posted September 27, 2021 Posted September 27, 2021 So, do we get an answer of what happened?
larryllix Posted September 27, 2021 Posted September 27, 2021 1 hour ago, chevykid said: So, do we get an answer of what happened? We are using a cloud service. This is typical and a good lesson for us all. Sounds like it was Amazon servers. Mine was intermittent in Ontario but came back last night.
johnstonf Posted September 27, 2021 Posted September 27, 2021 Hi Larry,I'm a tad confused... who is the 'we' you speak of?Sent from my SM-N950U1 using Tapatalk
johnstonf Posted September 27, 2021 Posted September 27, 2021 Is isy cloud some kind of club we belong to and all have responsibilities in? What were the lessons I was supposed to learn? Sounds like maybe a configuration problem, but if this is open source, how do we work on it?Sent from my SM-N950U1 using Tapatalk
MrBill Posted September 27, 2021 Posted September 27, 2021 (edited) 48 minutes ago, johnstonf said: What were the lessons I was supposed to learn? The lesson is that it may not always work... Always have a plan B. I haven't heard what the root cause with some portal accounts was yesterday (It did NOT affect all accounts... mine never went offline). I was communicating with Michel yesterday morning on an unrelated ticket and added to the end of a reply do you know Portal is down for a lot of people with a link to this thread. In the next reply he told me it wasn't down but they were working with several people via tickets that were having issues. Shortly thereafter he replied above. TWO DIRECT LESSONS: Have a plan B to turn your lights on or off. Relying on Alexa only isn't smart. When I started my system originally I had a rule "no outside or internet dependencies", that got relaxed at some point, but not having or knowing what plan B is in case alexa (or other internet or cloud dependency) is temporarily down or offline. It happens. with Alexa there are countless dependencies from your internet connection itself, to up-stream providers, to amazon itself, not to mention the ISY portal itself is running on AWS servers, at any given time something might not be in working order. First line for support is to open a ticket: support@universal-devices.com sure it might be interesting to post in the forum also to see if others are having the problem. The first place it will be seen by multiple UDI people is if you open a ticket. While Michel reads the forum from time to time, it's not his priority. Tickets will get more attention than the forum anytime. Edited September 27, 2021 by MrBill 2
Athlon Posted September 27, 2021 Posted September 27, 2021 (edited) @johnstonf I'm not going to presume to answer for larryllix, but my guess is the lesson is we are all at the mercy of any cloud services we use (in this case AWS). That's all he meant to say, I believe. UD is relying on AWS for the portal, and obviously by extension so are we. When AWS has issues, so will we. Thank goodness it does not happen very often. I use Gmail and even they 'went down' for a short time last week. It happens - that's the lesson. There is not much you can do about it except to minimize the number of cloud services you rely on. I'm sure UD did everything they could if AWS needed them to repair the issue. EDIT: I'm mentioning AWS because someone else did in this thread - I don't know exactly what caused this issue. I believe larryllix's comment was directed toward cloud services such as AWS, however. Edited September 27, 2021 by Athlon 1
johnstonf Posted September 27, 2021 Posted September 27, 2021 Well of course, I don't even know how you could set this all up without having local to start with because that's how you set it up and then you add on the cloud so I'm still not sure what lesson I need to learn. Of course I can change it locally but I can't teach everybody how to use interfaces and have apps on their phones etc. Sent from my SM-N950U1 using Tapatalk
romeo_echo Posted September 27, 2021 Posted September 27, 2021 2 hours ago, MrBill said: The lesson is that it may not always work... Always have a plan B. I haven't heard what the root cause with some portal accounts was yesterday (It did NOT affect all accounts... mine never went offline). I was communicating with Michel yesterday morning on an unrelated ticket and added to the end of a reply do you know Portal is down for a lot of people with a link to this thread. In the next reply he told me it wasn't down but they were working with several people via tickets that were having issues. Shortly thereafter he replied above. TWO DIRECT LESSONS: Have a plan B to turn your lights on or off. Relying on Alexa only isn't smart. When I started my system originally I had a rule "no outside or internet dependencies", that got relaxed at some point, but not having or knowing what plan B is in case alexa (or other internet or cloud dependency) is temporarily down or offline. It happens. with Alexa there are countless dependencies from your internet connection itself, to up-stream providers, to amazon itself, not to mention the ISY portal itself is running on AWS servers, at any given time something might not be in working order. First line for support is to open a ticket: support@universal-devices.com sure it might be interesting to post in the forum also to see if others are having the problem. The first place it will be seen by multiple UDI people is if you open a ticket. While Michel reads the forum from time to time, it's not his priority. Tickets will get more attention than the forum anytime. Thanks MrBill, good info. Quick question though: I've never contacted Universal Devices for support directly before... As a casual owner of a single ISY994i, am I allowed to open tickets by sending an email to that support@universal-devices.com also? Or do I need to be some kind of subscriber or developer or something to open a ticket? Cheers, Ralph
MrBill Posted September 27, 2021 Posted September 27, 2021 20 minutes ago, romeo_echo said: As a casual owner of a single ISY994i, am I allowed to open tickets by sending an email to that support@universal-devices.com also? Yes. UDI's support is golden too. If you own a UDI product by all means open a ticket if you need to communicate with support.
Michel Kohanim Posted September 27, 2021 Posted September 27, 2021 Hello all, Apologies for not getting back sooner. The root cause was the load balancer. At the highest levels, and for obvious reasons, we are continuously strengthening cyber security for all our infrastructure. We migrated the load balancers to have the connection end at the balancer rather than direct traffic to the web servers. There should have been zero impact. Alas, some got impacted. I apologize for the inconvenience. With kind regards, Michel 2 1
apostolakisl Posted September 27, 2021 Posted September 27, 2021 FYI, I was offline as well on Sunday morning. Two different ISY's.
johnstonf Posted September 27, 2021 Posted September 27, 2021 (edited) 4 hours ago, Michel Kohanim said: Hello all, Apologies for not getting back sooner. The root cause was the load balancer. At the highest levels, and for obvious reasons, we are continuously strengthening cyber security for all our infrastructure. We migrated the load balancers to have the connection end at the balancer rather than direct traffic to the web servers. There should have been zero impact. Alas, some got impacted. I apologize for the inconvenience. With kind regards, Michel Thanks for your honesty Michel! I figured it was either ddos or a config issue (i've done that with my load balancers too). At least it's something that can be fixed... this time... In the wild... I'm really concerned with the latest level of ddos attacks, MikroTik router vulnerabilities, etc, (that CloudFlare can't protect) so keep working on improving cyber security all the time... it's only getting worse with these, billions of IOT $5 devices with vulnerabilities, yada yada Steve Gibson... "someone somewhere has built, assembled, and is in control of a horrifically powerful botnet, unlike anything seen before. It consists of upwards of a quarter of a million MikroTik routers, nearly half of them with IP addresses in the United States. And as I said, I'm sure those brief 60-second attacks against Yandex weren't meant to harm them. They were meant to give the attackers some sense for the scale of this new offensive weapon they have created. So they must be feeling quite pleased with themselves now. Nothing can withstand 21 million web requests per second." Go to https://voip.ms and watch a company going from viable to on its knees in days... due to... yes... ddos ransom Edited September 27, 2021 by johnstonf
larryllix Posted September 28, 2021 Posted September 28, 2021 16 hours ago, johnstonf said: Hi Larry, I'm a tad confused... who is the 'we' you speak of? Sent from my SM-N950U1 using Tapatalk You, me, and anybody else that uses cloud services. We have to expect more problems as there is much more complexity and devices in the data paths that are beyond our control. (routers, cables, connectors, ISPs, servers for ISY Portal, Alexa servers, pathways back and forth between our ISYs and the two. Totally local and self dependent is the better way, and the cloud services may be very awesome but the lesson we may need to learn is, not to be totally dependent on cloud things for serious control. I try to always have a local switch or other control to backup important functions of lights and other devices. I avoid mobile dependence also. They are mostly cloud based data pathways and very inconvenient to use.
Michel Kohanim Posted September 28, 2021 Posted September 28, 2021 @johnstonf, Thank you. We are quite paranoid when it comes to security and, specifically, cloud services. And, as time progresses, things get more complicated. Just have to stay vigilant and keep ahead of these dark forces. With kind regards, Michel
Recommended Posts