warnings and errors after upgrade

johnnyt · May 28

at about 5:26 AM and 5:28 AM two of my three instances of the plug-in reported API Limit reached. They worked fine all day so I don't have them configured to poll too often. When I looked at the logs I see there were times that we flooded with warnings (up to 8 or more per second) and other errors.

Things like:

2024-05-28 05:29:04,383 Thread-700 urllib3.connectionpool WARNING  connectionpool:urlopen: Retrying (Retry(total=18, connect=30, read=28, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='ext-api.airthings.com', port=443): Read timed out. (read timeout=10)")': /v1//devices/2930162741/latest-samples

2024-05-28 05:29:05,365 Thread-697 urllib3.connectionpool WARNING  connectionpool:_put_conn: Connection pool is full, discarding connection: ext-api.airthings.com. Connection pool size: 10

2024-05-28 05:18:40,345 Thread-666 udi_interface      ERROR    pgSession:get: Connection error for https://ext-api.airthings.com/v1//devices/2930162741/latest-samples: HTTPSConnectionPool(host='ext-api.airthings.com', port=443): Max retries exceeded with url: /v1//devices/2930162741/latest-samples (Caused by ResponseError('too many 502 error responses'))

2024-05-28 05:28:04,528 Thread-547 udi_interface      ERROR    pgSession:response: Unauthorized: https://ext-api.airthings.com/v1//devices/2930162814/latest-samples: text:

I didn't see anything about API limit reached so it seems to be an erroneous state got reported to IoX (which triggered an email to warn me). Maybe the flood of warnings lead to that? Would it be a good idea to put in a "wait" between retries to avoid possible race conditions, or at least reduce the clutter in the logs? (Unless there's value in having all of those entries.)

Maybe there was a short break in service that affected 2 of the instances but not the third due to polling difference. The 3rd one did have a few warnings but maybe the break in service ended shortly after first few

I've attached my 3 debug logs (one was only set to "Warning" level. I've changed it to "Info" for next time)

Airthings-C_5-28-2024_62253_AM.zip Airthings-C_5-28-2024_62237_AM.zip Airthings-C_5-28-2024_61905_AM.zip