db2ace2 Posted November 19, 2018 Posted November 19, 2018 Hello Team, I have started having the ADMIN Console Crash and the Error Logs appear to me to have some kind of a Port Scan and possibly DOS attack happening??? I am not fluent with how to interpret the logs so I would like to ask if someone could take a look at this log and tell me what is causing this issue and possibly how to resolve it. [HTTP:0-21-5] 192.168.1.29:58587->80 <-- The Log shows several of these with Successive Port Numbers which appear to me to be some sort of a Port Scan??? I recently started using Network Resources again and have some WEBHOOKS updating the ISY Portal and thus this could be something to do with that? but I don't see any reason for this to be happening or know where to look for the answer as to how to solve it. I am on 5.0.14D Thanks Charles Seiler ISY Error Log.v5.0.14__Sun 2018.11.18 12.20.31 PM.txt
Michel Kohanim Posted November 19, 2018 Posted November 19, 2018 Hi Charles, Those come from 192.168.1.29. What is this device. Also, I would not consider this the source of the issue since they come in spurts and then go away. I suspect a node server of sorts. Can you please elaborate on what you mean by "Crash"? Does it just close? Do you get any errors? With kind regards, Michel
db2ace2 Posted November 19, 2018 Author Posted November 19, 2018 Hi Charles, Those come from 192.168.1.29. What is this device. Also, I would not consider this the source of the issue since they come in spurts and then go away. I suspect a node server of sorts. Can you please elaborate on what you mean by "Crash"? Does it just close? Do you get any errors? With kind regards, Michel Thanks for looking at this Michel. The 192.168.1.29 device is my windows10 PC where I run the ISY console. Admin console etc. it seems that when I start admin and clear error logs everything is ok. Then after running a bit these errors start and finally the admin console will lock up and not allow me to hit enter or make any changes. It does not show busy but I can’t do anything in it I then end the task and start it back up. I am not at home to test until tomorrow. I had been using som new Network resources to communicate with IFTTT and also with the HUE api. Everything there seems to work. I just remembered that I also changed the DNS for the ISY to be 8.8.8.8 but still at the same IP address of 192.168.1.15 port 8100. I cannot think of anything that would be pinging the ISY from the PC other than running the admin console and sometimes the portal or Alexa. My SmartThings hub and HUE hubs I always update using the iPhone not the PC. So I am not sure where this is coming from or to or why it chooses a different port each time. Have I possibly hit some resource limit perhaps?Thanks Sent from my iPhone using Tapatalk
Michel Kohanim Posted November 21, 2018 Posted November 21, 2018 Hi @db2ace2, Thanks for the update. I don't think you're reaching a limit. When you get to this state, does clicking anywhere on the Admin Console produce a beep sound?
J_Barrett Posted November 21, 2018 Posted November 21, 2018 Hi I am having the same problem [11/20 22:37:28.7] 3751742252 0 -170001 [HTTP:0-17-5] 192.168.1.2:56831->80 [11/20 22:37:28.7] [11/20 22:37:28.7] 3751742252 0 -170001 [HTTP:0-17]: POST[1]-->/services [11/20 22:37:28.7] [11/20 22:37:28.7] 3751742252 0 -170001 <s:Envelope><s:Body><u:GetSystemStatus xmlns:u=" [11/20 22:37:28.9] [11/20 22:37:28.9] 3751742252 0 -170001 [HTTP:0-17-5] 192.168.1.2:56832->80 [11/20 22:37:28.9] [11/20 22:37:28.9] 3751742252 0 -170001 [HTTP:0-17]: POST[1]-->/services [11/20 22:37:28.9] [11/20 22:37:28.9] 3751742252 0 -170001 <s:Envelope><s:Body><u:GetSystemStatus xmlns:u=" [11/20 22:37:29.2] [11/20 22:37:29.2] 3751742252 0 -170001 [HTTP:0-17-5] 192.168.1.2:56833->80 [11/20 22:37:29.2] [11/20 22:37:29.2] 3751742252 0 -170001 [HTTP:0-17]: POST[1]-->/services [11/20 22:37:29.2] [11/20 22:37:29.2] 3751742252 0 -170001 <s:Envelope><s:Body><u:GetSystemStatus xmlns:u=" Right now i show about 1100 ports in "Close Wait" between the ISY and the console Process Name Process Proto Local Local Address ID Port Port jp2launcher.exe 7064 TCP 54385 192.168.1.2 80 http 192.168.1.238 ISY Close Wait jp2launcher.exe 7064 TCP 54244 192.168.1.2 80 http 192.168.1.238 ISY Close Wait The console is only up about 20 minutes right now - If i let it sit over night it can consumes all the port and the console will lock up "showing socket open failed java.net.sockettimeoutexception" or java.net.SocketException: No buffer space (or some thing like that) and or the isy will crash. When i close down the console all the "Close Wait" will end. It seem any communication between the console & ISY opens up a new port. Healing the network really drive up the open ports i also tried it on 2 different machines running different versions of java Java (build 1.8.0_60-b27) 32 bit Java (build 1.8.0_191-b12) 64 bit i am using 5.0.14D & start.jnlp as the console Jim Barrett
db2ace2 Posted November 21, 2018 Author Posted November 21, 2018 2 hours ago, Michel Kohanim said: Hi @db2ace2, Thanks for the update. I don't think you're reaching a limit. When you get to this state, does clicking anywhere on the Admin Console produce a beep sound? Yes it does. Its as if the screen is locked or Busy and I cannot get in, so I end the task and start up again.
Michel Kohanim Posted November 21, 2018 Posted November 21, 2018 @J_Barrett, We'll definitely look into this. Are you saying that this happens ALL the time or after Z-Wave | Heal? @db2ace2, Have you tried the Esc or Alt-Tab? I suspect the login dialog is behind the Admin Console. With kind regards, Michel
J_Barrett Posted November 21, 2018 Posted November 21, 2018 Hi All the time - it appears any communication between the console & the ISY opens up a new port for each thing it does Jim Barrett
Michel Kohanim Posted November 22, 2018 Posted November 22, 2018 Hi @J_Barrett, Connections depend on the interval between each request. Keep-Alive is about 500 m.s. So, connections in and of themselves is not an issue. The problem is that the socket is in close_wait on your computer. Based on our code review and some experimentation, we know for a fact that ISY receives the FIN and closes the connection (otherwise, your ISY would run out of buffer space). The problem seems to be that ISY's ACK is not received by the computer. So, the computer's socket stays in close_wait forever. Do you have any firewall software, VPN, or any other packet filtering software on your computer? With kind regards, Michel
db2ace2 Posted November 23, 2018 Author Posted November 23, 2018 Hi [mention=3659]J_Barrett[/mention], Connections depend on the interval between each request. Keep-Alive is about 500 m.s. So, connections in and of themselves is not an issue. The problem is that the socket is in close_wait on your computer. Based on our code review and some experimentation, we know for a fact that ISY receives the FIN and closes the connection (otherwise, your ISY would run out of buffer space). The problem seems to be that ISY's ACK is not received by the computer. So, the computer's socket stays in close_wait forever. Do you have any firewall software, VPN, or any other packet filtering software on your computer? With kind regards, Michel Thanks Michel. I will meet to check the firewall. I will be out of town for my sons wedding thru Sunday so it will be early next week before I can check this out. Thanks for the follow up. Sent from my iPhone using Tapatalk
J_Barrett Posted November 23, 2018 Posted November 23, 2018 23 hours ago, Michel Kohanim said: Hi @J_Barrett, Connections depend on the interval between each request. Keep-Alive is about 500 m.s. So, connections in and of themselves is not an issue. The problem is that the socket is in close_wait on your computer. Based on our code review and some experimentation, we know for a fact that ISY receives the FIN and closes the connection (otherwise, your ISY would run out of buffer space). The problem seems to be that ISY's ACK is not received by the computer. So, the computer's socket stays in close_wait forever. Do you have any firewall software, VPN, or any other packet filtering software on your computer? With kind regards, Michel Hi No software firewalls only hardware one built into router - no vpn or filtering software A little more info on this. I have 2 isy having the same issue. i took my second one and reset it wiping out the config and then down graded it to 4.7.3. No issues with close_waits then upgraded it to 5.04 - no issues with close_wait upgraded it to 5.08 - no issues with close_wait upgraded it to 5.09 - close_wait problem is back and the problem remains for 5.10 - 5.11c - 5.14 Any thoughts ? Thanks JIm Barrett
Michel Kohanim Posted November 23, 2018 Posted November 23, 2018 @J_Barrett, Thanks so very much for the feedback. How many Z-Wave devices do you have? And, when you move to 5.0x, do you go through the migration process? With kind regards, Michel
J_Barrett Posted November 23, 2018 Posted November 23, 2018 6 minutes ago, Michel Kohanim said: @J_Barrett, Thanks so very much for the feedback. How many Z-Wave devices do you have? And, when you move to 5.0x, do you go through the migration process? With kind regards, Michel Hi I went from 4.7.3 to 5.0.14 straight. I have about 56 z-wave devices and 40 insteon just to be clear the close_wait can be reproduced on a ISY with no device installed Thanks Jim Barrett
db2ace2 Posted December 2, 2018 Author Posted December 2, 2018 On 11/19/2018 at 10:22 AM, Michel Kohanim said: Hi Charles, Those come from 192.168.1.29. What is this device. Also, I would not consider this the source of the issue since they come in spurts and then go away. I suspect a node server of sorts. Can you please elaborate on what you mean by "Crash"? Does it just close? Do you get any errors? With kind regards, Michel Hello Michel, Your Advice was spot on about the previous errors being Router Firewall Related. I switched the Orbi Router from IGMP Proxy Secured mode to Open Mode and the random sequences of Increasing Port Scan Requests appears to have subsided. At this time my Error Log has only the following errors remaining. Can you shed any light on what these may mean? at 2018/12/01 12:56:25 AM System -170001 [TCP-Conn] -1/-140002, Net Module Rule: 77 Sat 2018/12/01 12:56:59 AM System -170001 [UDSockets] HTTP:45 error:6 Sat 2018/12/01 12:57:04 AM System -170001 [UDSockets] HTTP:45 error:6 Sat 2018/12/01 12:57:09 AM System -170001 [UDSockets] HTTP:45 error:6 Everything seems to have continued to function but there could have been something missed when the failure occurred. I have several Cameras that send HTTP Requests to Update Vars in ISY and also I use IFTTT with Smartthings and HUE to update ISY Network Resources so these could be related to something with that kind if Network traffic. Thanks for your help! Charles Seiler
Michel Kohanim Posted December 2, 2018 Posted December 2, 2018 Hi Charles, I am so very glad to hear. You can safely ignore those errors: 6 means that the socket is in close-wait state and waiting for the client to send the final close. They will go away by themselves. With kind regards, Michel 1
nathan Posted October 18, 2019 Posted October 18, 2019 (edited) On 11/22/2018 at 11:12 AM, Michel Kohanim said: Hi @J_Barrett, Connections depend on the interval between each request. Keep-Alive is about 500 m.s. So, connections in and of themselves is not an issue. The problem is that the socket is in close_wait on your computer. Based on our code review and some experimentation, we know for a fact that ISY receives the FIN and closes the connection (otherwise, your ISY would run out of buffer space). The problem seems to be that ISY's ACK is not received by the computer. So, the computer's socket stays in close_wait forever. Do you have any firewall software, VPN, or any other packet filtering software on your computer? With kind regards, Michel Hi Michel, Sorry to hijack but I found this thread after trying to find others with my CLOSE_WAIT issue that I've been debugging. I'm not sure your explanation is right about CLOSE_WAIT. CLOSE_WAIT happens when the other side of the connection closes (sends FIN) and then the local kernel keeps the connection in a CLOSE_WAIT state until the local application calls close() on the socket. I think there is a bug in the admin console that was introduced at some point that is causing the application to leak connections and never close them. It seems like every device status report causes the admin console to issue a SOAP GetSystemStatus, which the console uses the response to change the flag indicator and block the UI if there is too much pending. The Insteon T-Stat makes this especially bad since when it is queried, it seems force the admin client to generate 10+ GetSystemStatus requests back to back, which then leak and get stuck in the CLOSE_WAIT state. @J_Barrettyou seem to be implying that your problem is resolved? Are you still getting connections stuck in the CLOSE_WAIT state? Thanks Edited October 18, 2019 by nathan
J_Barrett Posted October 18, 2019 Posted October 18, 2019 2 hours ago, nathan said: @J_Barrettyou seem to be implying that your problem is resolved? Are you still getting connections stuck in the CLOSE_WAIT state? @nathan No problem still exist
Recommended Posts