REST interface can be slow

[brandwidth]

I'm seeing this issue again - REST is slow, but some kind of caching means it's only slow "sometimes".

For example, http://192.168.1.9/rest/programs/?subfolders=true took >10s to respond, and Chrome complained that the XML wasn't valid: "Extra content at the end of the document". When I reload it, the ISY responds quickly and all is well.

Anything I can share here that'll help to debug this?

[brandwidth]

I added more logging to investigate this - a Python script makes a GET request and logs the round trip time whenever it's >1s.

Any thoughts on why the variance is so high? I'm not adding any new variables here - the ISY is in "steady state".

2020-03-19 11:26:59,185  /rest/vars/get/2 took 26.229227 s
2020-03-19 11:27:25,683  /rest/vars/get/1 took 21.483313 s
2020-03-19 11:28:02,481  /rest/vars/get/2 took 36.762416 s
2020-03-19 11:28:28,739  /rest/vars/get/1 took 21.242982 s
2020-03-19 11:29:29,586  /rest/vars/get/2 took 60.818626 s
2020-03-19 11:29:56,218  /rest/vars/get/1 took 21.617317 s
2020-03-19 11:30:06,656  /rest/vars/get/2 took 4.381121 s
2020-03-19 11:36:39,764  /rest/vars/get/1 took 1.236568 s
2020-03-19 11:39:29,366  /rest/vars/get/1 took 48.736331 s
2020-03-19 11:39:32,130  /rest/vars/get/2 took 2.735085 s
2020-03-19 11:40:37,765  /rest/vars/get/1 took 27.713981 s
2020-03-19 11:41:04,113  /rest/vars/get/2 took 26.317265 s
2020-03-19 11:41:30,130  /rest/vars/get/1 took 21.006414 s
2020-03-19 11:41:56,175  /rest/vars/get/2 took 26.016872 s
2020-03-19 11:42:22,522  /rest/vars/get/1 took 21.332925 s
2020-03-19 11:42:48,723  /rest/vars/get/2 took 26.171875 s
2020-03-19 11:44:37,447  /rest/vars/get/1 took 27.657693 s
2020-03-19 11:45:03,417  /rest/vars/get/2 took 25.938609 s
2020-03-19 11:45:29,726  /rest/vars/get/1 took 21.294449 s
2020-03-19 11:46:05,035  /rest/vars/get/2 took 35.279932 s
2020-03-19 11:46:34,973  /rest/vars/get/1 took 24.923365 s
2020-03-19 11:47:31,158  /rest/vars/get/2 took 56.153243 s
2020-03-19 11:48:23,084  /rest/vars/get/1 took 46.909512 s
2020-03-19 11:48:33,025  /rest/vars/get/1 took 4.043669 s
2020-03-19 11:49:25,868  /rest/vars/get/2 took 3.541272 s
2020-03-19 11:50:59,241  /rest/vars/get/2 took 1.206724 s
2020-03-19 11:51:50,860  /rest/vars/get/1 took 30.128240 s
2020-03-19 11:52:18,113  /rest/vars/get/2 took 27.223486 s
2020-03-19 11:52:44,127  /rest/vars/get/1 took 20.999266 s
2020-03-19 11:53:10,118  /rest/vars/get/2 took 25.961969 s
2020-03-19 11:53:36,368  /rest/vars/get/1 took 21.236130 s
2020-03-19 11:54:02,571  /rest/vars/get/2 took 26.173769 s
2020-03-19 11:54:41,961  /rest/vars/get/1 took 23.311470 s
2020-03-19 11:55:10,910  /rest/vars/get/2 took 28.919070 s
2020-03-19 11:56:12,239  /rest/vars/get/2 took 1.743591 s
2020-03-19 12:03:07,859  /rest/vars/get/1 took 1.229289 s
2020-03-19 12:03:09,157  /rest/vars/get/2 took 1.268243 s
2020-03-19 12:03:55,366  /rest/vars/get/1 took 1.226776 s
2020-03-19 12:05:46,565  /rest/vars/get/1 took 1.242285 s
2020-03-19 12:05:47,809  /rest/vars/get/2 took 1.206969 s
2020-03-19 12:08:15,245  /rest/vars/get/2 took 1.206420 s

 

[larryllix]

How are you inserting your credentials each time?

With the first time access, your browser will usually ask you to enter your credentials, after ISY refuses, due to "not permitted".  Every access time your browser will again get a refusal to respond due to security missing. Then your browser looks up every security it has ever been fed, and supplies them to ISY, without your knowledge.

You would need to pre-encrypt your credentials and also use a browser that is capable of it.

[brandwidth]

It's not being accessed using a browser - the python in question is:

loadTime = requests.get("http://192.168.1.9/rest/vars/get/1",auth=('<username>', '<password>')).elapsed.total_seconds()

 

[larryllix]

4 hours ago, brandwidth said:

It's not being accessed using a browser - the python in question is:

loadTime = requests.get("http://192.168.1.9/rest/vars/get/1",auth=('<username>', '<password>')).elapsed.total_seconds()

 

My bad.

Yeah, I was answering your first post were you did use a browser before I read the second post enough. 

You appear to be measuring the http turn around time. ISY may do the job but not respond very quickly. I have also found the RPi o/s to take some time to switch tasks at times.
If you can, try breaking the http function into a send and receive with
time.sleep(0.001)
between them to force RPi to allow I/o tasking.
 

On second thought. Those times are way too long.

[Michel Kohanim]

@brandwidth,

It all depends on:
1. Whether or not the time includes processing by your computer (i.e. making a connection)
2. What's ISY doing at that time

There are 3 HTTP tasks in ISY, so, if ISY is doing something with all three, then your request will wait. The best test case would be to do this on a factory reset ISY that has NOTHING else connecting to it (no port forwarding, no portal, no programs, ...).

With kind regards,
Michel

[brandwidth]

Michel - thanks for that super helpful background.

It's likely my ISY is getting swamped by more than 3 concurrent HTTP connections - I'll try to optimise things based on what you've shared.

[larryllix]

10 minutes ago, brandwidth said:

Michel - thanks for that super helpful background.

It's likely my ISY is getting swamped by more than 3 concurrent HTTP connections - I'll try to optimise things based on what you've shared.

28 and 56 seconds? Your ISY must be wearing hip-waders.  

Have you checked your battery device pages to see if they are locked with  green "1011" icons or programs constantly flickering colours?

[brandwidth]

I do have some battery devices which have updates 'permanently' pending - will try disabling automatic writes to battery devices and see if that has any impact - thanks for the tip @larryllix.

Update: looks like that was the issue (I'd not considered that it'd have an impact on HTTP)... I'm now seeing response times reliably under 200ms

[larryllix]

7 minutes ago, brandwidth said:

I do have some battery devices which have updates 'permanently' pending - will try disabling automatic writes to battery devices and see if that has any impact - thanks for the tip @larryllix.

Update: looks like that was the issue (I'd not considered that it'd have an impact on HTTP)... I'm now seeing response times reliably under 200ms

@Michel KohanimThose reserved battery device writes are dangerous. I had to learn the hard way also. Cached writes can tie up ISY and cripple it.