wmcneil Posted April 23, 2022 Posted April 23, 2022 When a vacuum is started from the Roomba app on my phone (in this case it was a scheduled start), in the ISY AC, the reported values are not updating (Running shows "Off", State shows "Charging", etc.) even though in the PG3 roomba node server log file I see updates happening consistent with the fact that the roomba is cleaning. Even after 10 minutes had gone by, the AC still did not report any changes. I tried a Query, and that had no effect.
wmcneil Posted April 25, 2022 Author Posted April 25, 2022 I tried sending a start command from the AC today, and that did not work (it was working when I tried it last week). I tried restarting the roomba node server, and now it appears stuck at the "initializing connection" to one of the roombas. There are error messages such as this in the log: 2022-04-25 15:25:00,831 Thread-1 udi_interface ERROR roomba:async_connect: Connection Error: timed out 2022-04-25 15:25:01,837 Thread-1 udi_interface ERROR roomba:async_connect: Attempting retry Connection# 0 One thing that I noticed last week when the roomba node server did appear to be communicating with my i7+ is that it labeled its node as something like a "basic" roomba, which does not seem correct. This seems as broken as it was when I last tried to get the roombas working a year or more ago. Not complaining, just stating that with my three roombas (960, 980, i7+) it continues to not work for me. I am wondering if there is a fundamental problem with trying to control the roombas both from the AC and from the roomba phone app? - this is just brainstorming, not clear if this is an issue or not.
wmcneil Posted April 25, 2022 Author Posted April 25, 2022 Murphy's law is enforcing itself on me in a big way today. I finally figured out that one of my vacuums was not getting sufficient connectivity with the charge pads on the base to obtain a proper battery charge. After I thoroughly cleaned the pads, and put the roomba back on the base, the roomba phone app is now reporting "battery charging, too low to start" for that roomba. I will have to wait until the battery charges up to do more testing, but for that one roomba I was definitely getting strange and inconsistent results due to the contacts being dirty and the battery being low, including what I now realize was connectivity problems with the wifi network. (Strange thing is that the clean light was turning on, and the vacuum seemed to behave as if it was rebooting when I forced a reboot, but it finally occurred to me to try cleaning the contacts - they were a little dirty. )
bpwwer Posted April 25, 2022 Posted April 25, 2022 I mainly just ported the existing PG2 version to PG3 and did some work to try and make the initial discovery work better. However, the library being used to interact with the Roomba's is all reverse engineered as there is now official API available to interact with them. If you send me a log package along with some details of what you were trying and when so that I can correlate what's in the log to what you're seeing, I'll see if I can make it work better. 1
wmcneil Posted April 25, 2022 Author Posted April 25, 2022 Bob, thank you. After thoroughly cleaning the pads on all three of my roombas and bases, and making sure the batteries were all charged up without question, I have done some testing and I now have consistent connectivity and response to commands from the AC. Also, whether the roomba is started from the phone app, or the AC, the status in the AC seems now to be tracking correctly. The only thing I notice that is a limitation is that the end of the node name for my i7+ roomba is "basic roomba", whereas my 960 node name ends in "series 900 roomba", and my 980 node name ends in "Roomba 980". So the i7+ is not having all its features recognized, for instance in the AC it does not have a "action on full bin", or "passes" variable. The lack of these features is not causing me any problems at present, just something to be aware of.
bpwwer Posted April 25, 2022 Posted April 25, 2022 If you turn on debug logging there should be a line in the log that lists the capabilities: "Capabilities: Position xxxx, CarpetBoost xxxxx, BinFullDetection xxxx" Those values determine what series it thinks to Roomba is; series 980, series 900, series 800 or none of the above.
wmcneil Posted April 26, 2022 Author Posted April 26, 2022 14 hours ago, bpwwer said: If you turn on debug logging there should be a line in the log that lists the capabilities: "Capabilities: Position xxxx, CarpetBoost xxxxx, BinFullDetection xxxx" Those values determine what series it thinks to Roomba is; series 980, series 900, series 800 or none of the above. I set debug mode and restarted the node server. Based on the values printed out in the log file, when the node server starts up and prints out the saved capabilities ( INFO roomba-poly:handleRobotData: Loading saved robots), interestingly, all three roombas have the exact same list, and exact same values which I have summarized below: 5ghz 1 area 1 binFullDetect 2 bleDevLoc 1 dockComm 1 eco 1 edge 0 hm 0 lang 2 langOta 0 log 2 maps 3 mc 1 multiPass 2 oMode 10 ota 2 pmaps 5 pp 0 prov 3 sched 2 svcConf 1 team 1 tileScan 1 tLine 2 wDevLoc 2 Then the node server apparently queries the vacuums, and some capabilities are now reported to be different (and have different names:) 2022-04-26 09:47:22,258 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Getting capabilities from hRoomba2ndFloor 2022-04-26 09:47:22,259 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Capabilities: Position: True, CarpetBoost: False, BinFullDetection: True 2022-04-26 09:47:27,910 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Getting capabilities from hRoomba1stFloor 2022-04-26 09:47:27,911 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Capabilities: Position: False, CarpetBoost: False, BinFullDetection: False 2022-04-26 09:47:37,283 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Getting capabilities from hRoombaBsmnt 2022-04-26 09:47:37,284 Thread-1 udi_interface DEBUG roomba-poly:addNodes: Capabilities: Position: True, CarpetBoost: True, BinFullDetection: True I have attached the log file. Roomba_4-26-2022_94804_AM.zip
bpwwer Posted April 26, 2022 Posted April 26, 2022 I tried updating it to add support for the i7 (which is currently providing the same support as the 980). Try version 2.0.4 and let me know.
wmcneil Posted April 26, 2022 Author Posted April 26, 2022 I installed 2.0.4 and restarted the roomba node server. The node server status in the pg3 dashboard shows "disconnected." I tried rebooting polyglot, I tried rebooting IoP, nothing helped. I notice in the roomba node server page, it states the following "If you need to re-discover devices, use the "Discover" button in the UI to start the discovery process. This will clear any exising devices and start from an empty list." I'm not seeing a discovery button in either the PG3 gui or the AC gui? Roomba_4-26-2022_45539_PM.zip
bpwwer Posted April 26, 2022 Posted April 26, 2022 I almost always break something when adding enhancements. Version 2.0.5 should work better and actually start. 1
wmcneil Posted April 27, 2022 Author Posted April 27, 2022 16 hours ago, bpwwer said: I almost always break something when adding enhancements. Version 2.0.5 should work better and actually start. I restarted the Node server, it updated to 2.0.5, and now the i7+ shows the same list of attributes in the AC as the 980. One little anomaly I noticed, the graphic symbol in the AC next to the i7+ (hRoomba1stFloor) is now a light bulb, as opposed to whatever you call the symbol next to the other two roombas:
wmcneil Posted April 28, 2022 Author Posted April 28, 2022 20 hours ago, bpwwer said: Ok, thanks. I fixed it in 2.0.6 I updated, and the symbol is now the same as the other two. Thanks!
wmcneil Posted April 30, 2022 Author Posted April 30, 2022 Paul, I did some more testing with my i7+ this morning and roomba node server 2.0.6 . With no roombas running, I opened the AC on IoP. It reported Running==off and State==Charging for all three vacuums, which is correct and as expected. I started my i7+ from the Roomba app on my phone. There was no change in the AC. I waited about 5 minutes, and the AC reporting had still not updated. I tried doing a query from the AC. After another 5 minutes the AC had still not shown any updates. Log attached. You can see the startup of the i7+ (which I did from the Roomba phone app) being reported by searching on 08:54:29,891 in the log file, and subsequent entries show that the roomba continues to make updates about its progress to the node server. So not clear why the AC is not updating, is there something about the fact that the roomba was started from the phone app that is related to why the AC is not updating? Roomba_4-30-2022_90456_AM.zip
bpwwer Posted April 30, 2022 Posted April 30, 2022 All three Roomba's were configured in the node server during this test right? The log doesn't really tell me much. I do see something changing state, but it seems to change state for less than 5 seconds. However, that may just be the other Roomba's state mixed with the i7's I can't really tell. The node server is supposed to update the state in the AC every 5 seconds (so there can be a delay between commands given via the app and the state changing in the node server) based on the default shortPoll time. Neither the underlying library being used to communicate with the Roomb's nor the node server itself output any info to identify which Roomba it's getting updates from or when it updates the node server state. Thus I can't really tell what is really happening. I'll have to add some debug info into the node server so we can see what's really happening. One thing that concerns me is that all the messages in the node server log indicate they are coming from "Thread-1". The node server is multi-threaded and I would expect each Roomba to be doing things on a separate thread. But because of the way the underlying roomba library works, that may not be true for the actual communication with the Roomba.
bpwwer Posted April 30, 2022 Posted April 30, 2022 Ok, I added some debug messages to 2.0.7 that should allow me to better trace what is happening. If you can run the same test again, but make sure you have debug log set for the node server, I'll see if I can figure out what's happening.
apostolakisl Posted April 30, 2022 Posted April 30, 2022 I have an i7 (not sure about the plus part) and other than not getting the x,y,theta values, it seems to work.
fahrer16 Posted April 30, 2022 Posted April 30, 2022 Trying to use the app and the node server simultaneously will likely give sporadic results. The roomba hosts an mqtt server for local communications that has historically only allowed a single connection, so if the app is connecting locally the node server might get kicked or at least stop getting updates. I used to run my Roombas on a separate VLAN so the app wouldn't be able to connect locally (cloud should still work and wouldn't interfere with the node server comms). An additional option is restricting internet access from that VLAN so the Roomba's can't update their firmware automatically and break the reverse-engineered protocol the node server relies on (which also means the app won't work). I think one of the Roomba firmware updates a few months ago killed the tracking, causing the x, y, and theta values to no longer get reported.
apostolakisl Posted April 30, 2022 Posted April 30, 2022 52 minutes ago, fahrer16 said: I think one of the Roomba firmware updates a few months ago killed the tracking, causing the x, y, and theta values to no longer get reported. I suspected this was the case since I know it worked a couple weeks ago and I saw an update just happened about a week ago. I also keep my roomba on a vlan that is less secure and punched a hole in the firewall so it can talk to Polisy. The new PG3 roomba node server is challenging since it autodiscovers roomba only, no manual option, so I had to temporarily move polisy to the roomba vlan. I can firewall roomba from wan, but without the app, you can never make any setting changes to roomba. Maybe I never will need to?
wmcneil Posted May 2, 2022 Author Posted May 2, 2022 (edited) On 4/30/2022 at 11:34 AM, bpwwer said: All three Roomba's were configured in the node server during this test right? The log doesn't really tell me much. I do see something changing state, but it seems to change state for less than 5 seconds. However, that may just be the other Roomba's state mixed with the i7's I can't really tell. The node server is supposed to update the state in the AC every 5 seconds (so there can be a delay between commands given via the app and the state changing in the node server) based on the default shortPoll time. Neither the underlying library being used to communicate with the Roomb's nor the node server itself output any info to identify which Roomba it's getting updates from or when it updates the node server state. Thus I can't really tell what is really happening. I'll have to add some debug info into the node server so we can see what's really happening. One thing that concerns me is that all the messages in the node server log indicate they are coming from "Thread-1". The node server is multi-threaded and I would expect each Roomba to be doing things on a separate thread. But because of the way the underlying roomba library works, that may not be true for the actual communication with the Roomba. I reran a test with 2.0.7 . Log file attached. I made sure all roombas were docked and charging, then restarted the roomba node server. Search in the log for "2022-05-02 09:40:01,306" for the restart of the node server. I started up just the i7+ roomba from the phone app. (It was also the case in the last test that I posted results for that only the i7+ roomba was running.) This time the IoP AC was reporting the status of the i7+ correctly. I suspect what @fahrer16 posted above about the roomba mqtt server consuming the one local connection, and precluding the node server from connecting, is happening sporadically, and causing the problems (sporadically) when the phone app is used locally. (Not clear why sometimes this happens, and other times I can initiate a run from the phone app and the AC does report correctly - Is the Roomba phone app sometimes not using the local connection for unknown reasons? It would be nice if the roomba app had an option to only use the cloud. ) There are lots of settings and controls that can be configured/used only with the phone app, so never using the phone app is not a good option currently. With regard to putting the roombas on a separate vlan, and modifying firewall rules so the node server can talk to them, maybe this is a possible workaround for me, BUT, it will cost a lot of time, of that I am sure. I have a pfSense router , which is highly configurable, but also quite complex, so it also always costs me a lot of time to change anything. (I have bi-directional OpenVPN running between my primary home and a vacation home, with different subnets at each location, and it took many weeks of work to get that configured and working properly.) Roomba_5-2-2022_94545_AM.zip Edited May 2, 2022 by wmcneil
bpwwer Posted May 2, 2022 Posted May 2, 2022 I think @fahrer16explaination makes sense. He knows a lot more about this node server than I do as he originally wrote it. I just did the updates to make it work with PG3.
fahrer16 Posted May 3, 2022 Posted May 3, 2022 Most of what I know comes from the project that the node server uses: NickWaterton/Roomba980-Python: Python program and library to control iRobot Roomba 980 Vacuum Cleaner (github.com). From the various troubleshooting I've done, it seems like the first to connect to the Roomba locally wins. For example, if the connection from the node server is alive and well and then the app is used, it seems like it connects via the cloud and leaves the node server alone. If the app is able to connect locally, it seems like the node server isn't able to and is out of luck. I've wondered if maybe the local mqtt connection times out or if there isn't much traffic while the roomba is idle and it gives the app the opportunity to "steal" the one and only local connection. You can tell there's a lot of guessing going on, which is always the challenge of using reverse-engineering unsupported API's. Definitely pros and cons associated with isolating the roombas vs keeping them open to use the app at any rate.
wmcneil Posted May 3, 2022 Author Posted May 3, 2022 Thinking some more about a use model to deal with the only-one-local-connection issue: Let's explore using only the node server for day-to-day scheduling/running of the roombas. (I am making the assumption here that if the phone app is not configured with any automatic schedules for the roomba, there is no possibility of the phone app consuming the local connection. A firmware update could use it, not sure how often that occurs, and will assume it is infrequent enough to be ignored for the purposes of this discussion.) What features/functions are missing from the node server? Here are ones that come to mind: For models with the Clean Base (charging base with integrated bag that allows roomba to self-empty its full bin by returning to clean base, and then potentially continue cleaning): node server does not have an "empty bin" command. Most of the time the Roomba will automatically empty its bin and continue cleaning without requiring an explicit independent empty bin command. Occasionally, when in doubt about whether the Roomba bin is really empty, I have used the "empty bin" command on the Roomba app, but that is very infrequent for me. For models that support mapping, and are capable of cleaning only a specific room, and also capable of avoiding areas using the mapping interface on the phone app: Clearly the setup for the mapping can only be done with the phone app, but that setup occurs quite infrequently. I virtually never use the "clean only one room" feature. Alexa interaction: I don't use this feature at all, so I can't comment here. So maybe for my use model, the "use only node server for day to day scheduling/running" will work. I should mention my primary motivation for wanting to use the node server: The roombas can set off my motion detectors on my alarm system. So I want IoP programs running that detect if any roomba starts while the alarm system is armed, and immediately stop the roomba(s). If the phone app/scheduling is used to start the roombas, the problem with this approach is that sometimes the node server can not communicate with the roombas (due to only-one-local-connection problem), so there is no way to achieve the goal.
apostolakisl Posted May 5, 2022 Posted May 5, 2022 I appear to have a similar issue. Roomba was running, but AC showed it was in charger. I closed/reopened AC and it is not populating Roomba. Roomba log in PG3 shows a continuous flow of data as you would expect from a running Roomba without any errors. I tried using commands from the AC and Roomba does not respond nor does the log show anything except the flow of info from Roomba. In other words, it appears the the NS has lost its connection to ISY. I then restarted the NS (just the NS, not polyglot or polisy or isy) and all is now working correctly. I do not know why the NS would lose its link to ISY. I haven't made any changes to ISY or any NS's since the last reboot.
wmcneil Posted May 5, 2022 Author Posted May 5, 2022 4 hours ago, apostolakisl said: I appear to have a similar issue. Roomba was running, but AC showed it was in charger. I closed/reopened AC and it is not populating Roomba. Roomba log in PG3 shows a continuous flow of data as you would expect from a running Roomba without any errors. I tried using commands from the AC and Roomba does not respond nor does the log show anything except the flow of info from Roomba. In other words, it appears the the NS has lost its connection to ISY. I then restarted the NS (just the NS, not polyglot or polisy or isy) and all is now working correctly. I do not know why the NS would lose its link to ISY. I haven't made any changes to ISY or any NS's since the last reboot. Was the roomba started from the roomba server, or from the node server?
Recommended Posts