More Speed!

[johnnyt]

I would gladly pay a premium to get more speed when the console loads and programs are saved. While I might not have wanted to pay more to get into this ISY game and I’ve always found it a little slow, now that I have close to 400 programs (probably 1/4 are “flags†set up before getting into variables), I find it painful to wait for everything to load – especially when I want to quickly check something – as well as during program saving, backups, restores and log viewing (well, loading).

 

Since a recent upgrade to 3.1.16 it also appears I might be running into system busy problems that weren’t happening in 3.1.11 and prior, even though I’m not using any of the ELK functionality added since 3.1.10/11. (am considering going back to 3.1.11 but support becomes a problem when one does not not keep up with latest code base.) This one is still being troubleshooted at the time of writing this, but if this issue could be solved with a faster CPU in the ISY, there would be some urgency to offering this, IMO. See http://forum.universal-devices.com/viewtopic.php?f=25&t=7693&p=58853#p58853 for details/outcome.

 

Maybe a faster CPU and/or Gigabit NIC in the PRO version? (sign me up now if that’s already the case today!)

 

In the mean time, could part of the speed problem be blamed on network issues? I have Gigabit NIC in my PCs and my router. I’m pretty sure the PC CPU doesn’t matter that much because I get roughly the same performance from three machines of different vintages (6 yr old AMD2200, 4 yr old Core2 Duo, and 1 yr old i3), all running Java 1.6 update 29.

 

I do love the ISY otherwise. And the support has been great. That's why I'm approaching 400 programs and still have plans for probably double that over time if/when I can move everything over from HomeSeer - although if I'm hitting a wall that can't be knocked down performance wise, I'm going to start questioning that plan.

[Michel Kohanim]

Hi johnnyt,

 

I think we are mixing a few issues here which might be related but the conclusions drawn are perhaps not correct.

 

 

I would gladly pay a premium to get more speed when the console loads and programs are saved.

Let's stop it right here. I can understand longer loading for 400 programs vs. 1 but how many programs are you saving at a time? If one or two, then there's surely a program or a few programs that are staying busy/running ALL THE TIME. This explains everything else in the rest of your post.

 

Basically, every time ISY runs a program, it puts a lock around a few resources so that they are not changed while the program runs. This happens in nano seconds. Now, if you have many programs in loops after loops calling one another, sending INSTEON commands, which generate 10s of events (say you turn on/off a scene) which are evaluated and in turn cause other programs to turn on/off scenes, programs, then obviously:

1. You will see System Busy a LOT (ISY still has to communicate with INSTEON devices)

2. You will have much slower loading of programs (resources are locked)

3. You will have much slower backup/restore (file system is locked)

 

So, what I recommend are:

1. Find a pattern/time that the system busy is at its peak

2. Go to Program Summary

3. Find all programs in Running state

4. Stop them one at a time

 

Repeat above, till you find 1 or 2 programs that are causing all this.

 

With kind regards,

Michel

[johnnyt]

Thanks for the reply, Michel, and for the insight into contributing factors to performance problems.

 

I'm a little concerned the solution to the problem might be to not run too many programs, especially when my plan/hope is to increase my use of ISY, which will mean adding and running more programs concurrently.

 

I did do some testing to see if I could find something I was doing horribly bad, or to see if maybe I could understand/frame the problem differently. Below is what I found.

 

Allow me to first confirm that I rarely save more than a handful of programs at once and often only 1 or 2 at a time, particularly when I’m just trying to do a quick tweak (when more speed would be particularly welcomed). I’ll also confirm that I do have one program that runs all the time, however disabling it did not improve performance in my testing, except perhaps when the system was under load (more on that later).

 

The one program I have that I’ve been running all the time (7x24) for several month now is this one below. It monitors the DSCLink updated heartbeat variable that gets changed every 60 seconds.

 

If

$dscHeartbeat is not 0

 

Then

$dscCommsDown = 0

Wait 30 seconds

$dscHeartbeat = 0

 

Else

Wait 1 minute and 1 second

$dscCommsDown = 1

 

 

While I do have other programs that will run for extended periods of time they only wait to turn something off or on. While some such programs ran during my tests, none ran for the whole duration of a test run. Hopefully the “wait†command uses very few resources anyway and that isn't an area I need to watch in my programming approach – please let me know if that isn't the case.

 

I figured I would use the backup command to test performance under different load conditions, including with and without the above program running. I assume that's a pretty good way to test performance (and be a good benchmark?) since the backup function has to access all resources – again, please let me know if this isn’t the case.

 

To confirm the relative load ISY was under I opened the event viewer, sorted on Last Run Time and refreshed the list after the backup finished to see what, if any, programs ran. I also watched as programs ran (that I could see on the screen.)

 

In the best case (no programs ran and no events occurred), the backup took a little over 3 minutes. That already seems too long to me and I would happily pay to reduce the time it takes since I do backups almost every time I make a change. In the worst case, it took about 5 minutes. Most of the time, the backups took between 3 1/2 minutes (no or low load) and 4 1/2 minutes (some to moderate load).

 

It turns out the above program running under no or low load did not make a difference. It does appear to have made things worse under load but it's hard to say that with certainty.

 

In the worst case, the following notable activities occurred:

• - the above program was running,
  - motion detection in one area was going on every 10 - 40 seconds
  - a program that "waits" after affected motion detector sends OFF was starting/running/stopping periodically (2-3 times max)
  - an HVAC related program that is triggered by a call for heat via an IOLinc Sensor caused the following to happen once:
  
  • query 8 IO Linc relays via a scene
    turn on a couple of duct dampers

- other HVAC related programs were evaluated but not run (i.e. false with no Else actions to run)

 

From where I sat, while the above activity appeared to add 10-20 secs of processing (including waiting), 1 1/2 to 2 minutes were added to the processing of the backup.

 

If you tell me that more processing power can’t improve on this, I’ll have to believe you. The problem I see is that there will be a lot more work for ISY to do if I am ever able to move my HVAC thermostat, temperature/humidity sensors, and forecast related functions over from HomeSeer. With administrative tasks taking exponentially longer to complete as my program footprint grows, I can only re-iterate that I would happily pay for a faster processor.

 

Any other performance improving advice would be welcome too, of course.

[Michel Kohanim]

Hello johnnyt,

 

Thanks so very much for the detailed response.

 

First of all, having 100s of programs should not have that much of an impact on performance and thus I would not worry about it. Secondly, the program that runs 24x7 should run in milliseconds (no INSTEON commands are issued). And finally, by programs calling programs causing performance issues is IF AND ONLY IF the Then section is doing too many things with INSTEON which is basically the performance bottleneck.

 

All this said, I think there's probably something wrong with your system: I have never seen backup take such a long time even in a system with 100s of devices and 100s of programs. Can you please:

1. Take the SD Card out of the unit and let me know the brand ... also - and if you have a computer with SD Card reader - put the SD Card in your computer, copy everything in the /CONF directory and then save it in a different directory. Does it take more than a few seconds?

2. How is your ISY connected to your router and router to your computer?

 

Unless there's something else that is using the file system (perhaps you can put an always false condition in My Programs folder to disable all programs and recheck), I think the problem might be a defective SD Card.

 

With kind regards,

Michel

[johnnyt]

It's good to hear this isn't normal. It isn't just backups that are slow. Everything is slow. Console startups, loading programs, backups, restores, log viewing, saving programs, etc. I did nonetheless try setting program folder to a false condition to see and it didn't improve backup speed.

 

SD Card brand is Sandisk, model is Ultra II. I don't have an SD card reader to test copy speed elsewhere.

 

I was wondering just a couple of weeks ago whether or not I should worry about having an SD card as a backup since it's my understanding they have a shorter life span than hard drives. Is this true? How long can they be expected to last in an ISY? I could buy one to test whether or not this is the problem then keep it as my backup. What brand/model/size would you recommend?

 

ISY is connected directly to my router. FWIW, I did also do some testing with my LAN under load by playing a Windows Media Center recording via an XBox 360 and copying a 70GB VM to another machine. Although my VM copy throughput went down a bit, LAN load had little impact on backup performance in my tests.

[Michel Kohanim]

Hi johnnyt,

 

You do not need a backup SD Card: everything you need (in case of a crash) is already in your backup zip file.

 

We have had units working under heavy operations for more than 3 years with the same SD Card. SD Card's life expectancy should be of no concern.

 

How old is your unit? I can certainly send you an SD Card.

 

With kind regards,

Michel

[johnnyt]

You do not need a backup SD Card: everything you need (in case of a crash) is already in your backup zip file.

I didn't mean keeping a spare sd card to store my backup; I meant so I have a replacement on hand and can be back up quickly in the event of an SD card failure. I do this with other key devices. For example I keep a spare PLM on hand. Also KPL, switchlinc, etc. It's already paid off a couple of times.

 

I got the unit in the fall of 2010.

[Michel Kohanim]

Hi johnnyt,

 

I would be delighted to send you a different SD Card to test with if you don't already have one you can try.

 

If you wish, please send me your shipping address to support@universal-devices.com.

 

With kind regards,

Michel

[johnnyt]

thanks for the offer. I'm fine to get my own SD card and keep the one I have now as an emergency backup since it does still work.

 

What speed class of card should I get, i.e. will a class 10 (10MB/s) make any difference or is it enough to get a class 2 (2MB/s)?

[Michel Kohanim]

Hi johnnyt,

 

ISY works with all ... SDHC is currently the best as far as capacity and speed.

 

With kind regards,

Michel

[johnnyt]

ISY works with all ... SDHC is currently the best as far as capacity and speed.

Yes, I realize that. Hmmm, I guess you're either being careful to not appear to have a favorite or you don't want to divulge internal speed specs.

 

Maybe someone else can answer? Will faster SD card speed make any noticeable difference? Up to what speed?

 

Am only using 13MB of space so capacity is the least of my worries...

[Michel Kohanim]

Hello johnnyt,

 

My preference is SanDisk ... we do not use SDIO; we use SPI interface which means that the maximum speed (regardless of what SD Card speed you get) is 10Mbps.

 

With kind regards,

Michel

[johnnyt]

Update:

 

A couple of days ago I picked up a 4GB Class 6 SDHC card I ordered last week on sale for $5, including USB reader. Not the fastest available but a sweat deal that also allows me to read the cards on another machine.

 

Here's some quick data on the two cards I have (using HD Tune 2.55):

 

• old SD: Sandisk Ultra II 512MB: 8.6 MB/s avg - no damaged blocks
  new SD: Transcend 4GB: 18.5 MB/s avg - no damaged blocks
  

 

I'm confused a little because the Class 6 cards are supposed to be 6MB/s, yet it's showing 18.5 MB/s avg throughput (read and write).

 

Also I'm realizing all this speed talk is moot since – unless it was typo – ISY max speed is 10 Mega bits per second, which means only 1.25 MB/s.

 

Whatever is happening behind the scenes, here's what I observed moving to the new SD card:

 

• - speed for restores is dramatically better (about 3 1/2 mins instead of 8 1/2) and no failures during the "clearing old configuration" step (see

http://forum.universal-devices.com/viewtopic.php?f=27&t=7745)
 
- speed for backups is a little over 2 minutes. I did see a 2 minute backup just before replacing the SD card but I also saw many 3-4 min and a 5 min backup before that. Haven't seen the latter yet with the new card.
 
- initial GUI load seems a little quicker. It takes 5-6 secs and I would think it was pushing 10 secs before, although I didn’t time it.
 
- loading programs takes 2-3 secs to start (between the time I click the tab and the "loading programs" message comes up) and about 12 secs to complete. This is perhaps marginally better than before.
 
- saving programs seems a little quicker. The system is still busy for about 30 secs after the actual “saving†message finishes but that seems better than before too
 
- new card didn’t fix the apparent performance problem when I have to send a notification and update a state variable in the same program. (for details see http://forum.universal-devices.com/viewtopic.php?f=25&t=7693&p=59009#p59009)

 

It does appear some of the performance problems were related to the SD card. In particular when extensive writing to the card was being done, e.g. a restore. This make me wonder if there’s an issue related to lack of garbage collection on the SD card. It’s my understanding this is known to cause notable writing performance problems on solid state disks once they fill up with deleted sectors. I wonder if I had reformatted the old card (or forced a Window 7 TRIM command on it) to refresh it whether it would have improved performance. I may yet try that but at this point I have a new card so it's not a problem. Plus would the performance hit related to this result in SD throughput of less than the ISY max of 1.25 MB/s?

 

While I’m glad to have seen performance improvements from where things were, I would still happily pay to get more speed. In more concrete terms the ideal would be sub second loading and saving times as well as backups that take less than 20 secs. However, since I realize PC-like performance would be too expensive, I would be happy with something that brings one closer to 5-6 secs to load everything, 10 - 15 secs to save, and less than 1 minute to backup 100 devices and 500 programs.

 

In the mean time, if there's anything else I can do at my end to get better performance, please let me know.

[Michel Kohanim]

Hello johnnyt,

 

Thanks so very much for the update.

 

I am still concerned with saving programs taking too long. That's NOT normal. Saving programs should take a second or less unless ISY is busy. I still think there's something going on either in your programs or your network.

 

How's your ISY connected to your network? Is there a switch in between? If so, what's the brand?

 

With kind regards,

Michel

[junkycosmos]

I too had been in search of trying to speed up admin consul related work. Backups take take my system around the same and I 2nd your interest in this area. Few weeks back I did replace my original SD card with new sandisk and found some slight speed increase. Still not sure though if the increase was new card or fresh install on formatted card.

 

My last discovery has also been temperature relayed. When my ISY was under 30 degrees F I found that the Ethernet communications slowed down, around 20 and comm would be nearly non usable. Moving my ISY to a 40+ degree eng has done wonders for fixing this. More details here viewtopic.php?t=7896

 

Now temp issues completely aside I just timed my backup: 3 mins 26 sec

 

1st launch of programs tab after open of admin console: 12 sec

 

Typical saving of a few program updates takes: roughly 3 secret program updated

 

Interested to see where this topic goes. Pls let me know if I can provide anything useful here too

 

J

 

 

Isy99ir pro

323 programs/most are tiny, one 1-3 typically active running

184 device on my network list (bunch of kpls so they count as 8 etc)

80 scenes

[Michel Kohanim]

Hi junkycosmos,

 

Just to be clear here, based on your input, you have at least:

323 (programs) + 184 (devices) + 120 (database) + 80 (scenes) ~= 700 objects + our configuration/management objects that are backed up and restored. During backup/restore, FOR EACH object, ISY has to wait for a lock. So, if any devices are programs are running which use those object, backup/restore will wait till that object is no longer busy. So, the basic question is how long should backing up a 1000 objects through the network take.

 

Saving programs and loading programs uses the same exact locking mechanism and thus subject to the same constraints. Perhaps, as a test, you can disable all your programs, unplug the PLM, and then try saving a program and/or backup/restore. That would be a good measurement for performance.

 

Having performance issues at low temperatures, on the other hand, is not normal and something I would like more information on. I'll post my questions on the other post you have.

 

With kind regards,

Michel

[johnnyt]

Hello johnnyt,

 

Saving programs should take a second or less unless ISY is busy.

 

How's your ISY connected to your network? Is there a switch in between? If so, what's the brand?

My ISY is connected directly to my Netgear FVS336Gv2 gigabit router. FWIW, one of the machines I use, a W7 PC, goes through a wired Netgear gigabit switch to my router. The other, a wirelessly connected W7 laptop, goes through a D-Link Access Point (no routing) connected to a D-Link 100Mb switch connected to my router. Both take the same amount time to do things in the GUI.

 

To be clear the file saving itself only takes a second (for one program anyway). What takes ~30+ secs is waiting for the system to stop being busy right after the save. I did some more checking and the latter delays any subsequent saving and slows navigation down but does not prevent me outright from navigating around. At the end of the day this is not a big deal. (Perhaps only displaying the "system busy" message if someone tries to save again will hide this otherwise benign activity?). Am more interested in faster loading and backup times.

 

Thanks.

[Michel Kohanim]

Hi johnnyt,

 

Thanks so very much for the information. We shall obviously look for any opportunity to improve the performance and your feedback is quite important in determining what we should look at.

 

So, in short, after you save a program, the system might become busy for a while. We'll start from there.

 

With kind regards,

Michel

[johnnyt]

I may have found the reason for my system going busy after saving a program. After pausing my Homeseer ISY plug-in saving a program does not result in the system going busy.

 

Disabling the HS plug-in, however, did not improve initial load time and program load time. In fact the programs took a whopping 27 seconds to load in my 1st test after having disabled the plug-in. A sort by last run time indicated that only one program (the one mentioned before) had run during the time the programs were loading. In my follow up test, the loading of programs returned to the 12 secs or so reported earlier.

[Michel Kohanim]

johnnyt,

 

I think we are chasing a ghost here. How many programs do you have?

 

If you have 100s of programs and if you keep changing them, then with every change, ISY has to regenerate the total number of bytes that have to be sent back to the UI. This is why the initial load was that long. On top of this, all the locking that goes on for every object adds to the time. But, 24 seconds? That does not make any sense. Do you have more than 300 programs?

 

With kind regards,

Michel

[johnnyt]

Am at 389 programs, 126 total devices, 53 unique devices, 73 scenes, 13 network resources and 54 variables (according to HS ISY plug-in)

 

I can see that a lot of programs will add to loading time but I'm sorry I don't see how the frequency of my changes would be related to initial program load time. I can't make any changes to programs until there are loaded.

[Michel Kohanim]

johnnyt,

 

Every HTTP response requires the total size of the response to be known before hand. So, when you click on the Program Detail/Summary tab, Admin Console will request ISY to send back all the programs. If you had not changed any of your programs, then ISY uses a cached number for the total size and the load time is faster. As long as programs are not changed, ISY keeps using the cache. When a program is changed, the next load time will go through the same calculation to find the total size.

 

So, you have about 400 programs and let's assume it takes 25 seconds to load them ... that translates to 0.06 second per program. Even though I think 25 seconds is a little high and that something else is contributing to this, 0.06 second per programs is pretty impressive. I would be very interested in finding "servers" out on the Internet that are as efficient.

 

With kind regards,

Michel