Runnning two ISY's in one house

[johnnyt]

Am thinking of offloading some of my nearly 650 programs (170 variables, 71 devices/156 nodes) to a second ISY to improve performance when I want to make changes and/or when the system is busy. For most of the devices I plan to offload there would be a clean break - the devices will only be linked to one of the two PLMs I would have - but for a few devices I would like to link them to both PLM's.

 

Is it do-able? What are the pitfalls, if any, of doing this? I mean both from the perspective of having two PLM's in one house in general, and from having devices storing link records generated by two different PLM's.

 

Also, specifically with respect to having two PLM's, from what I understand each PLM's commands would be echoed/retransmited by the other PLM. Is that right? Is that a source of potential problems and/or can it actually be helpful (like adding an access point)?

 

Any info would be appreciated.

[LeeG]

Having multiple PLMs in the same house is not a problem. I have several PLMs that are active at the same time. I also have two ISYs that are currently active at the same time (not sharing devices).

 

The PLM is like any other Insteon device in that it repeats Insteon messages it receives that are not specially sent it. The PLM chip in a PLM device is the same PLM chip in other 120v Insteon devices.

 

Sharing devices will be a problem. Neither ISY knows about the other which nets down to multiple HA applications trying to manage the links in the same device. That has not worked with combinations such as HouseLinc and ISY trying to manage the same devices. Link records are deleted, overlaid, etc.

 

There is a user on this forum that is using the Network Module to send REST commands between two ISYs but devices are not being shared.

[johnnyt]

Thanks, LeeG.

There is a user on this forum that is using the Network Module to send REST commands between two ISYs but devices are not being shared.

Searching for "two ISY and REST" I found a recent relevant post and a whole bunch of other irrelevant posts (which wasn't a big surprise given the prevalence of search words I used). The one relevant post (viewtopic.php?f=26&t=11761&hilit=two+isy+and+rest) does not really explain how to do this with REST beyond needing the network module and network connectivity. If you happen to remember anything about this topic (like the user who did it) that might help me find more info.

 

I was walking through my migration plan in my mind and figured I would want/need to do the following things (someone pls correct me if I'm wrong):

 

1. restore ISY1 backup to ISY2 (ensuring appropriate firmware version alignment first - or will that happen with the restore?)

2. restore new PLM connected to ISY 2 (or should I/can I do the next step first?)

3. delete devices and programs on ISY2 that are not intended for ISY2

4. delete devices and programs on ISY1 that are not intended for ISY1

5. restore each device via the ISY that is now supposed to control them

 

My questions are:

 

1. for step 3 will deleting devices on ISY 2 muck things up for ISY 1 in any way? (from what I understand, no, but would like to confirm)

 

2. If I want to work this out on the bench / in the lab - same house - before I deploy things into "production", I know I can put a filterlinc on the PLM to block power line signals but I don't think I can stop the RF signal from getting picked up by one of my many access points. Do I need to worry about that? From what I understand, devices will NOT respond to PLM2 until AFTER step 5 above. Is that right? Also, I know there would be extra insteon traffic that could interfere with normal operation of ISY 1 but will collision avoidance logic in the two PLM's work to reduce the impact of that?

 

3. If I wanted to stop PLM RF transmission, is there a way to do that by opening up the PLM (with or without voiding the warranty)? Would just putting it in a metal box work?

 

4. Am I missing anything I should be aware of?

[Michel Kohanim]

Hi johnnyt,

 

I am not sure why you would even entertain this idea as it would make your system much more complex to understand and manage.

 

Is the main reason the load times?

 

With kind regards,

Michel

[johnnyt]

Is the main reason the load times?

Hi Michel,

There are a few reasons for me to entertain this idea. Mainly it's because ISY does not have the horsepower needed to handle my peak workloads. The 60-90 sec GUI loading is certainly one of the workloads but there are other times when a high workload causes problems. And it's one of those things that isn't getting better as both the number of programs grows and the firmware footprint increases.

 

1. At any given time I have 6-12 programs running (mostly waiting but doing something every now an again) and it's not uncommon to see about 10-20 programs that last ran in the same second. Every 15 mins or less I have a program that queries about 20 IOLinc nodes (described in the 5th paragraph of this post viewtopic.php?f=27&t=11907). During the 7-10 secs this runs in some case it can cause maybe 40 programs to at least evaluate and a few of them to do something. Whenever this program runs the GUI is not able to keep up. It doesn't show what's going on as it's happening and I'll get some sort of socket failure message if I try to do anything. I've also returned to my computer to find one of those socket failure messages on the screen, i.e. things got overwhelming on their own. (I do leave the GUI running to mitigate the loading time issue.)

 

2. In addition to the workloads triggered by ISY programs, my external temp/humidity sensor solution (viewtopic.php?f=78&t=9840) now updates about 18 state variables every minute, between 1 and 2 secs apart each. Some of the state variables are in 20-30 programs and these will, at minimum, evaluate. ISY also needs to update or respond to communications coming from HomeSeer, DSCLink and ISY Data Logger (by io_guy). All four of these external subscribers make real time demands of ISY and are completely outside the control of ISY or any insteon collision avoidance algorithm. Multiple times a day both DSCLink and ISY Data Logger indicate they can't update their respective heartbeat variable. Often, DSCLink is not able to pass the activation of a alarm panel connected motion sensor to ISY and a program to turn on a light does not run, or runs very late (directly impacting the WAF). Sometimes (usually when I'm loading or saving programs) the HomeSeer ISY plug-in has to activate it's watchdog timeout to reconnect.

 

3. Notifications - which I often rely on for troubleshooting in the absence of a program log - are often delayed because they are queued up to run separately and end up running well after they were triggered. A good design idea when processing resources are scarce, except that they will often report programs ran False instead of True (when they were actually triggered). This means one can't rely on them to know exactly when a program ran, and one has to work around the issue if one wants to know with certainty that an else statement actually ran. I put in a 2 sec wait and ensure there's a command after the wait - sometimes having to insert a "dummy" command - since waits don't run if they are the last command. Moving to a 2nd ISY will not fix this; am just mentioning it as additional evidence that ISY hardware is not powered sufficiently to handle high workloads, at least mine).

 

4. In my case, I actually thought it might slightly improve manageability (although the jury is still out on that). About half my programs are to handle HVAC operations (see viewtopic.php?f=48&t=10174). Although there are a couple of exceptions where a KPL that controls something HVAC related also controls a light, I think I would be able to have a mostly clean/clear line between what the two ISYs would control.

 

5. Finally, we are likely moving in 12-18 months and I'm thinking of leaving the HVAC automation, which is unique to the house and has a bunch of highly embedded stuff (hardwired sensors, duct dampers, HRV, etc), and taking the other automation with us. I'm thinking having that separated out (and stable well before we start the buy/sell/move process) would be a good thing.

[Michel Kohanim]

Hi johnnyt,

 

Thanks so very much for the details. I must say that above and beyond the GUI load issue, none of the other scenarios seem to be that resource intensive. For instance, getting socket open error while IOLinc program runs is probably not related to resources but things that should NOT be happening such has tight loops.

 

In any case, you are welcome to try but knowing a little about your configuration I think you are going to have more problems than solving them.

 

With kind regards,

Michel

[johnnyt]

getting socket open error while IOLinc program runs is probably not related to resources but things that should NOT be happening such has tight loops.

I don't think I have any tight loops, but maybe I don't understand what you mean by that.

 

Here's an example of a loop as I understand the term. The 6-12 programs that are running at any given the time look like this.

 

Edit: Program 'Query Countdown':

If
       Program 'Query HVAC Devices II' is False
Then
       Repeat Every  1 minute 
          $sHVAC.Query.Count += 1
Else
  - No Actions - (To add one, press 'Action')

 

Below are three very typical programs that are triggered by changes in some state variables or IOLinc node statuses: (the first two work together) - Do any of these qualify/cause "tight loops"?

 

Edit: Program 'Fan High Speed Off - Temp Sensor':

If
       $sFanHighSpeedManual is 0
   And $iUpDownTempDiff < 100
   And $iUpDownTempDiff is not -999
   And $sTemp.MasterBedroom is not 0
   And Status  '1-MISC (Non Lighting) / HVAC / Kitchen.F-FanHigh' is On
   And Program 'Fan High Speed Off - Temp Sensor II' is False
   And Program 'HVAC - AC Call Flag' is False

Then
       Wait  3 seconds
       Run Program 'Fan High Speed Off - Temp Sensor II' (Then Path)

Else
  - No Actions - (To add one, press 'Action')

Edit: Program 'Fan High Speed Off - Temp Sensor II':

If

Then
       Disable Program 'Fan High Speed Off - Temp Sensor'
       Wait  15 minutes 
       Run Program 'Fan High Speed Off - Temp Sensor III' (If)
       Wait  3 seconds
       Enable Program 'Fan High Speed Off - Temp Sensor'
       Run Program 'Fan High Speed Off - Temp Sensor II' (Else Path)

Else
  - No Actions - (To add one, press 'Action')

 

Edit Program 'Furnace Damper ON':

If
       Status  '1-MISC (Non Lighting) / HVAC / Furnace Rm Damper' is Off
   And Status  '1-MISC (Non Lighting) / HVAC / Fan Low Speed' is Off
   And Status  '1-MISC (Non Lighting) / HRV / HRV Low Speed' is Off
   And Status  '1-MISC (Non Lighting) / HRV / HRV High Speed' is Off
   And (
            Status  '1-MISC (Non Lighting) / HVAC / Fan On' is On
         Or (
                 Status  '1-MISC (Non Lighting) / HVAC / Sensor - AC Off - Fan Low Spe' is Off
             And $sPowerFailure is 0
            )
       )
Then
       Wait  15 seconds
       Set '1-MISC (Non Lighting) / HVAC / Furnace Rm Damper' 100%
Else
  - No Actions - (To add one, press 'Action')

[LeeG]

I suspect a loop issue would not be within a single program. A program changing the conditions that program is checking is pretty obvious. More along the lines program A triggers/runs program B that triggers/runs program C that changes something that causes program A to trigger/run again in a loop through B and C that ends when some other condition changes, temp, time, some combination of conditions, etc.

 

Have you flow charted the Program structure, what program calls what program under what condition. With that number of programs it does not seem possible to have all the interrelationships memorized.

[johnnyt]

I'll look for programs that call other programs that call themselves, etc. but the first two code samples above are a good and very typical example of programs that call another program in my setup. The programs that call a second program only call ONE other program and are the ONLY ONES that call that specific other program. As you see in the example in my previous post, the second program doesn't loop back into the first program, the second program DISABLES the first one AND the first one is prevented from running when the second one is running by the condition

And Program 'Fan High Speed Off - Temp Sensor II' is False

Often I have to use a second program to mimic an "else if" although in this and many other cases it's to workaround the fact that my temp variables change every minute. With my temp variables changing every minute (sTemp.MasterBedroom in the above example) I have to use a second program when I need to wait anything more than 59 secs - in this case a wait 15 mins done in the second program would never complete in the first program. I would also mention that, as shown in the example, I always put in a wait command, between 3-15 secs (depending on what the program is supposed to do) to allow the variables/node statuses to stabilize.

 

So I believe I've been careful in thinking through each program and its impact on the overall state of things. I think the problem is that with 20 IO Linc nodes being checked within about 10 secs and multiple programs monitoring a number of them along with data coming in from other places - most notably my temp sensor solution - there will be many programs evaluated in a short period of time - what perhaps behaves like a "tight loop" but is really just a high workload, IMO.

 

In the mean time, could I go back to my questions?

 

I was walking through my migration plan in my mind and figured I would want/need to do the following things (someone pls correct me if I'm wrong):

 

1. restore ISY1 backup to ISY2 (ensuring appropriate firmware version alignment first - or will that happen with the restore?)

2. restore new PLM connected to ISY 2 (or should I/can I do the next step first?)

3. delete devices and programs on ISY2 that are not intended for ISY2

4. delete devices and programs on ISY1 that are not intended for ISY1

5. restore each device via the ISY that is now supposed to control them

 

My questions are:

 

1. for step 3 will deleting devices on ISY 2 muck things up for ISY 1 in any way? (from what I understand, no, but would like to confirm)

 

2. If I want to work this out on the bench / in the lab - same house - before I deploy things into "production", I know I can put a filterlinc on the PLM to block power line signals but I don't think I can stop the RF signal from getting picked up by one of my many access points. Do I need to worry about that? From what I understand, devices will NOT respond to PLM2 until AFTER step 5 above. Is that right? Also, I know there would be extra insteon traffic that could interfere with normal operation of ISY 1 but will collision avoidance logic in the two PLM's work to reduce the impact of that?

 

3. If I wanted to stop PLM RF transmission, is there a way to do that by opening up the PLM (with or without voiding the warranty)? Would just putting it in a metal box work?

 

4. Am I missing anything I should be aware of?

[LeeG]

I don't see a problem with the actions to ISY 1. There are all kinds of problems when moving to ISY2.

 

First the Restore Modem (PLM) will change all the device link records to point to the PLM connected to ISY2. That damages the ISY1 environment.

 

Deleting the ISY1 devices from the ISY2 configuration will have PLM errors because the link records do not exist in PLM2. No idea what will actually happen. Don’t know of anyone who has tried to split one configuration into two ISYs.

 

At some point a Restore Modem (PLM) will be needed on ISY2 to build PLM2 and change the device link databases for ISY2 devices to point to PLM2. Might be able to accomplish the same thing with a Restore Device to the ISY2 devices. Again, no history/experience for what you want to do.

 

Folks who have tried to isolate a PLM from the powerline have not always been successful with a single FilterLinc. Two may be needed. Putting the PLM into a solid metal box should block RF but power has to enter the box which may allow some RF to leak.

 

Bottom line is I have many doubts about the procedure. I guess the worst is both ISY configurations are not functional in the end requiring the ISY backup to be restored to ISY1 and the PLM and devices restored.

[johnnyt]

Hmm, not pretty. Looks like if I wanted to go with a second ISY, I'd have to configure it from scratch.

 

I broke up my worse high workload offender IO Linc query scene into two, spread the call to them apart by 12 seconds and made my programs fit into either the 7-12 sec space between the 1st and 2nd query or wait 7+ secs after the second query. I also stopped querying sensors separately after what I learned recently about IOLinc queries (viewtopic.php?f=27&t=11907). It's actually too bad (although understandable) that an IO Linc query hits both relay and sensor status because I only use 3 of the possible 11 sensors but have to live with the traffic - and apparent high degree of ISY processing resources it consumes - for 9 sensors I don't care about.

 

In my definition of the perfect world, it would have been so much less work to just throw another CPU at it, as I can do with my HomeSeer VM... (and would still today happily pay for here - see viewtopic.php?f=7&t=7768&) even if that's not the most elegant / best way from an engineering point of view.

 

This won't reduce the poor performance related to loading the GUI, backups, restores, etc. and it remains to be seen if it will eliminate all the non responses to external system events or noticeable delays of visible events like turning a light on/off, affecting the all important WAF.

[Michel Kohanim]

Hi johnnyt,

 

Throwing more CPUs will not at all help with INSTEON communications which MUST be serialized and tracked. To make matters more interesting, INSTEON does not have a transaction ID so, for every request, we literally have to jump through hoops to make sure the response is actually the response to the command sent.

 

GUI load/etc., unless you are using HTTPS, I think additional CPU resources is not going to help at all since the bottleneck is the file system read operations (one file at a time).

 

With kind regards,

Michel

[johnnyt]

GUI load/etc., unless you are using HTTPS, I think additional CPU resources is not going to help at all since the bottleneck is the file system read operations (one file at a time).

I do remember (and found the post where you were) mentioning that

...we do not use SDIO; we use SPI interface which means that the maximum speed (regardless of what SD Card speed you get) is 10Mbps.

I stand corrected if instead of more CPU, I should be asking for the ability for ISY to leverage SD speeds. I don't know how much you would have to charge to go with SDIO at your end but I can get an 4GB Class 10 (10 megaBYTES per second) microSD card for about $10.

 

I have no doubt you jump through hoops to make things as solid as they are. I wish I didn't have to jump through hoops at my end, particularly if it could be fixed with a bit more horsepower - whether that means faster CPU, RAM, disk, NIC or a combination. If it can't, it can't. Or if it's a case of it could but it simply won't, well, thems the breaks, as they say. In the mean time, I'll work with what I have and can get. At this point, I don't think it will be adding a second ISY so it currently rests on the changes I mentioned in my last post.

 

Thanks for your replies.

[Michel Kohanim]

Hi johnnyt,

 

Not even SDIO is going to help since the problem is NOT bandwidth as each file is less than 1K.

 

With kind regards,

Michel