Dual Band PLM Communicating Poorly with Older (I1) Devices

[Brian H]

ELA, You are correct. The power line transmitter and receiver sections are not in the partial schematic of the early Icon and ApplianceLinc. New ones are completely different. No more 120 Volt AC relays and other differences.

 

You can see it in the photos of the older hardware Icon and ApplianceLins.

http://efundies.com/guides/insteon/hard ... page_1.htm

[ELA]

IndyMike,

 

Sorry if my questioning came off as snide.

When people state things that I do not understand I question them.

 

A while back you and I shared some PMs where you commented about how your Insteon install worked so well and that you did not understand why others performed poorly.

Your desire was to help others understand their issues. You got serious, good for you. Perhaps a little too serious?

 

Best of luck on your quest. I will comment no further in this tread unless requested to do so.

 

Again I apologize if I offended you.

[IndyMike]

ELA,

 

Thank you for the post regarding the Icon coupling. I don't know how I got it in my head that the device used transformer coupling. I may have been seeing what I wanted to see...

 

Regardless, that was an excellent catch that fundamentally changes my search for "root cause" on the I1 zero cross timing issue.

 

Some time back you had questioned whether "excessive" Insteon voltage levels could affect communications. The 2412S schematic does not appear to have hardware filtering on the zero cross input. As you noted, the Insteon signal could (should) itself cause variations in the zero cross trip point. The PLM appears to deal with this in some manner - it does not exhibit the phase variation that the other I1 devices have. Firmware??

 

Aside from that, I've borrowed an isolation transformer from a friend to retry the phase measurements. At the same time, an old field issue has cropped up at work. Things will be slowing down significantly as I have customers from three different companies in house to help in resolving the problem.

 

Unfortunately, I also took those problems out on you. I really wish I could retract my previous post. Since I can't, the best I can do is apologize and encourage you to continue questioning things that don't appear to make sense.

 

IM

[IndyMike]

Your testing has made me think again about something I've always wondered: why Insteon seems so much more susceptible to impulse and amplitude noise than it really ought be. The PLL sync really should perform much better in such an environment.

 

But that requirement for tight zero-crossing synchronization suggests the possibility it isn't just simple AM interference fouling the transmissions as we usually assume. But instead high impulse noise creating localized, anomolous jitter of the zero-crossing, then causing either the transmissions to be mistimed relative to the PLM or the reception at the PLM to be ignored.

 

Are you able to tell if the harmonics are being back-fed from your own residence or coming in off the pole?

 

Hello ergodic,

 

I have never understood how such a simple zero crossing detector works for the powerline, let alone rf synchronization. I'm assuming that there's a little "magic" happening under the hood of the PIC. Rather than trying to crack that nut, I'm trying to understand the differences in timing of the I1/I2/PLM devices. After ELA corrected my recollection of the Icon schematic, I am again struggling to figure out the differences.

 

Impulse noise could certainly be a problem if it occurred at the 60 Hz zero crossing. Typically, it does not. Inductive loads drawing current on startup can also be a problem. Normally these are transient problems.

 

My incoming power is a bit squared off as shown below. The black trace is the AC powerline and the blue is a pure sine reconstruction based on the true RMS reading from my Fluke.

 

My scope can see harmonics (3rd, 5th, etc) but, at 8 bit resolution, it can't really resolve any changes. I'm pretty sure this is incoming from the transformer. None of my loads seem to affect it.

[ELA]

Hi IndyMike,

We all have our moments, not to worry. Thanks for your reply, and invite back I hope...

 

I enjoy thinking about why Insteon communications may be malfunctioning but get frustrated with a lack of detail required from SmartLabs in order to best understand the issues.

My feeling is that they know of the issues but are not sharing.

 

To this point we are forced to speculate. While I prefer to test and understand it is a daunting task without knowledge of the firmware.

Here is my present theory (speculative mind you):

 

I think we agree that Zero Cross detection has too much component tolerance variances to be the absolute reference.

I did some calculations a while back and can easily see variances of 55 usec or more in the detection threshold from tolerance variances, especially when adding in a worst case CTR of the opt-isolator. This based on the 2412 and 2413 detection circuits from early Insteon schematics and some circuit tracing.

( I wanted to add that my limited circuit tracing of the i1 keypadlinc was thorough enough to exclude transformer coupling, however I could not identify an opto-coupler. So there are very definite differences in zero cross detection circuits over releases).

 

My best guess is that each unit uses its own zero cross reference to begin an internal timer. The timer function has potential to be very accurate due to the sort length of instruction cycles of the Micro-controller running at ~22Mhz.

Each unit could then stop and record that timer value when it begins to receive a message. This then provides an accurate reference value for calculating when to begin its reply or simulcast.

 

In this case even though there may be a significant difference between one units zero cross ref. to another's they could still be "synced".

Of course if the local zero reference is varying from cycle to cycle it could still be upset. Also if the micro-controller were to become very "busy", or code oversights allowed the internal timers to be erratic at times, these could also adversely affect the synchronization.

 

I was hoping earlier that you would expand on your harmonics suspicion. I understand the concept that lower level harmonics could affect the rise time of the wavefront, and thus zero cross threshold/timing. But I think if this affected each unit is would not matter, only if some were subjected to it and others were not?

That is why I thought an isolated network would possibly rule that out (even though lower harmonics can pass through the transformer).

 

I sincerely hope that there may be a local power quality issue that can correct the i1 device issues but I suspect evolution of the fittest will rule out.

[ergodic]

IM:

 

It's never been clear to me exactly how the PLM handles the dual RF and powerline signaling together, except that it seems to be much more complex than the general descriptions I've read.

 

I've been dealing the last few months with some odd "X-factor" Insteon problems. The kind that can't be explained by noise or signal-suckers or really any stable logic at all.

 

Replacing the 2413S PLM with a new V.93 greatly reduced the problems for a while, and apparently there are changes directed to some of those comm issues.

 

However, about 10 days ago things started to revert back. No explanation, nothing had changed. Scenes partly failing even though I could never get an ISY scene test to fail. Random "unable to communicate..." pop-ups from the motion sensors, etc. My scope showed a solid signal and very little noise at the PLM plug. Reprogramming the PLM and devices didn't help.

 

The last straw was walking into the garage and seeing the MS light flashing. Unable to communicate with the PLM. A PLM that's 10 feet away.

 

So... mostly out of frustration, I moved the dual-band PLM over to the filtered side of power, forcing it to communicate RF-only. (I have a passive coupler and also two APs dropped directly out of the panel which is also in the garage.)

 

Remarkably, not only did that immediately resolve all the comm problems, but the system is now performing better than it ever has. No kidding.

 

I use a whole-house ISY query as a general metric of how things are working. 50-60 seconds has been typical when things were working well. The query is now ~40 seconds with no repeated ACKs in the log and no comm errors from anything, and all scenes seeming to work perfectly since that one simple change.

 

How long will this last? Dunno - I may be posting next week that it's all fallen apart again. I don't think so, but who knows?

 

Why exactly did this work? Can't really explain that either. (But too bad the PLM firmware isn't open source, it would be interesting to see I think.)

 

However I do infer from all this that the dual-mesh system is decidedly a work in progress. Especially as it relates to integration with older / i1 devices. And that taking powerline comm. competition completely out of the dual-band PLM equation can definitely help.

[IndyMike]

Hello ELA,

 

Sorry for the Hiatus, my customers have kept me hopping.

 

Hi IndyMike,

Here is my present theory (speculative mind you):

 

I think we agree that Zero Cross detection has too much component tolerance variances to be the absolute reference.

I did some calculations a while back and can easily see variances of 55 usec or more in the detection threshold from tolerance variances, especially when adding in a worst case CTR of the opt-isolator. This based on the 2412 and 2413 detection circuits from early Insteon schematics and some circuit tracing.

( I wanted to add that my limited circuit tracing of the i1 keypadlinc was thorough enough to exclude transformer coupling, however I could not identify an opto-coupler. So there are very definite differences in zero cross detection circuits over releases).

 

My best guess is that each unit uses its own zero cross reference to begin an internal timer. The timer function has potential to be very accurate due to the sort length of instruction cycles of the Micro-controller running at ~22Mhz.

Each unit could then stop and record that timer value when it begins to receive a message. This then provides an accurate reference value for calculating when to begin its reply or simulcast.

 

In this case even though there may be a significant difference between one units zero cross ref. to another's they could still be "synced".

Of course if the local zero reference is varying from cycle to cycle it could still be upset. Also if the micro-controller were to become very "busy", or code oversights allowed the internal timers to be erratic at times, these could also adversely affect the synchronization.

 

We're on the same page with the timer referenced zero crossing. Something on my powerline will (at times) cause I1 devices to incorrectly calculate this timer value. This incorrect value has minimal impact when dealing with 0 Hop communications. When responding to 3 hop communications the error is multiplied and can cause the PLM to ignore the response.

 

I2 devices, accesspoints, and PLM's appear to be relatively immune to this error. The I1 devices are also immune when simulcasting (or at least the errors are minimal).

 

I was hoping earlier that you would expand on your harmonics suspicion. I understand the concept that lower level harmonics could affect the rise time of the wavefront, and thus zero cross threshold/timing. But I think if this affected each unit is would not matter, only if some were subjected to it and others were not?

That is why I thought an isolated network would possibly rule that out (even though lower harmonics can pass through the transformer).

 

I sincerely hope that there may be a local power quality issue that can correct the i1 device issues but I suspect evolution of the fittest will rule out.

 

My harmonics suspicion pretty much revolved around the I1 devices having a transformer coupled crossing detector. When you pointed out my error, I started looking elsewhere. I've acquired some new tools and have been monitoring the 60Hz and harmonics. Curiously, the harmonics actually decrease when the I1 devices encounter problems. At the same time, my 60 Hz fundamental begins to vary - hard to tell if this is amplitude or phase. I'll try to post some data when I get a better understanding of what is varying.

[IndyMike]

Hello ergodic,

 

It appears we've approached things in exactly the opposite direction. I've tried filtering the PLM to eliminate the powerline communication influence. When that didn't work, I disabled the RF on the PLM. Still no go. The fact that I still get comms errors on the filtered circuit is what has me focused on low frequency variations (I can't filter these easily).

 

What type of filter did you use for your PLM? Did you use a UPS with a sine output?

[ergodic]

No, it is a floating-power UPS so just raw AC feed-through. Unless it trips, then God knows what the waveform looks like. I've never looked but usually not pretty.

 

Interestingly your power looks almost exactly like mine, a little ragged clipping right before the top. I don't have a THD or SA meter, but it's hard to believe that would have any effect on the powerline communication.

 

I'm using a 15A Filterlinc. The UPS and PLM are both plugged in on that side now, but the PLM is not plugged through the UPS.

 

Since writing that last post I will say I continue to be astonished at the improvement in overall reliability of the system. I wish I had some explanation, but I suspect it is mostly related to firmware interactions.

 

Did you try this configuration with the V.93 PLM?

[Illusion]

IM,

 

Based on statements by ergodic, here is what I am going to do in the coming months:

 

I am going to put the troublesome i1 KPLs that helped send you down this dark hole back into service, albeit in different locations than original. I am going to try to get them to fail again in daily life like before. Then I am going to filter the PLM like ergodic has.

 

I will be most interested in seeing if it can correct my issues as well.

[IndyMike]

ergodic,

 

Thank you for that post. I had tried RF only coupling (PLM filtered) previously with very mixed results. I decided to revisit this after your post.

 

Test configuration:

1) 2413UH, Accesspoint, "Test KPL 0F.BF.A4" on filtered circuit (filterlinc)

2) Garage Entry KPL (09.8C.CD) at far end of house on Phase A.

3) 2nd accesspoint on Phase B, unfiltered side.

 

The results are rather definitive and repeatable. I still believe this is due to the I1 devices phase shifting. I believe what is happening here is that my phase B accesspoint is able to recognize the phase shifted signal when the PLM cannot. Switching to RF only allows the accesspoint to receive the phase shifted signal and pass it to the PLM.

 

I have a lot more data to post, but am short on time. Have to wake up in a few hours.

[IndyMike]

Illusion,

 

By all means, please do try this configuration. I am still trying to determine how many people may be affected by this phenomena. If you note an improvement, that may indicate confirmation.

[IndyMike]

Ergodic and Illusion,

 

Here's the data behind the RF VS Powerline testing. I performed 4 different tests using HL2 to evaluate the communication success rate. The results are presented with both PLM retries enabled and disabled. The "retries enabled" success rates will be closer to the ISY performance.

 

 

 

Lots of interesting things going on in the above.

• [*:3jcy1j8u]Test 1 Illustrates that I1 devices are able to communicate effectively through the Accesspoint. My "Test KPL" is close coupled to the PLM and not doing well communicating through the powerline. This is due to the "phase shift" phenomena that I had presented previously.
  [*:3jcy1j8u]Test 2 was conducted to illustrate that the "test KPL" could also communicate well via RF. This is the overall best run of the group with both KPL's communicating to the PLM via the Accesspoint.
  [*:3jcy1j8u]Test 3 is a bit perplexing. All I did was remove the filter between the PLM and the rest of the house. This is my "normal configuration". Since the accesspoint is able to relay the responses to the PLM via RF, one would also expect it to be able to simulcast the information on the powerline. According to the success rates above, something is interfering.
  [*:3jcy1j8u]Test 4 is essentially the same as test 1 with an Accesspoint in place of the DB-LL. The high success rates seem to indicate that the Accesspoint is able to receive and repeat the phase shifted communication from the KPL where the DB-LL is not able to do so. This was an epiphony for me. I had picked up a couple of DB-LL in an effort to improved communications of my I1 devices. My experiments in using these devices to "boost" signals on problem circuits were just plain confusing. Now I understand why.

 

I am not at all sure whether we are suffering from the same anomaly. I am hoping that the above configurations will allow you to determine if your symptoms are the same as mine.

 

Test Circuit1: PLM, Test KPL, and DB-LL on filtered circuit

Red - Powerline Communication

Green - RF Communication

 

 

Test Circuit2: Test KPL moved to "House Side" of filter (RF communication only)

[ergodic]

IM:

 

Test 3 flummoxes me a bit also. I need to get HL2 and see what's going on a little more under the hood.

 

Your #2 is - I think - the closest to the configuration I currently have: DB PLM on filtered side and APs and devices on the raw power side.

 

One of the devices I was having the biggest reliability problems with was an on/off KPL in the garage. I did replace that a while back with a DB KPL as one of my in-vain fix-it efforts. It perhaps slightly improved scene reliability on that device but not dramatically. But it is having no problem at all now in the current configuration with the PLM on the filtered side. I suspect if I put the old non-DB one back it would be OK too now, but I'm not going to mess with it.

 

I wouldn't be too quick to assume powerline interference, though it's the obvious assumption. I'm starting to be convinced that a lot of these "X-factor" issues are related to device firmware and inter-compatibillity. The hop-count timing 'shifting window' problem you see could be as simple as a programming flaw.

 

One of the things I have noticed is that my whole-house queries run much quicker and I believe most of that is due to PLM retries no longer happening much (which right now I can only infer from watching the commands in the log). There were occasional ISY repeat ACKs in the logs before, but a lot of times a single ACK queries would stall for a bit, and I attribute that to PLM retries. I do not see any of that now - the commands mostly just click right along.

[Illusion]

IM,

 

Great post with very informative drawings. Thanks. Indeed, the issues you are having seem similar to my V2.D KPL issues that you tried so hard to help me with.

 

Since replacing the 2 KPLs that all of a sudden stop being reliable when adjusting back-lights (due to adding other insteon devices on other circuits in the system), back-light adjustment on the two replacement KPLs is at 100% for the past 4 months.

 

I am going to use the shelved V2.D i1 KPLs as controllers for my newly acquired FanLincs as soon a the ISY supports it. I will include them it the daily back-light adjusting program. I suspect they will show failures, as they did before. Then I will jump the PLM to an isolated side of the power-line.

 

I kinda wish we had thought about this when I was struggling with the issue last year, but it seems so counter-intuitive.

 

My symptoms of my shelved i1 V2.D KPLs do seem to mimic your findings. Including test 3's anomalous high success rate. Kinda like when link table reads would be great for me, except when they weren't for no apparent reason. Which you attributed to noise at the time. I attributed it to the Insteon Dark Force.

 

IDF: you can't see it, you can't really measure it, so you can't really understand it. You know it is there because you can make tables and charts that show it exists. Sometimes buying identical newer devices fixes it. I cannot tell you how happy I am to not be trying to find the problem with those KPL communications. Replacing them was some of the best money I have spent on Insteon. I cannot believe I am going to go around again with them!

[IndyMike]

Hello ergodic,

 

I've been going back through my notes and realize that I've run this tests 1 and 2 previously. At that time I did not understand that the I1 devices exhibited a time/phase shift. In short, I used a DB-LL on the house side of the filter and the tests failed miserably. I now believe that the DB-LL is less tolerant of communication time/phase shifts much like the 2413 PLM's. This is also why adding DB-LL's to an I1 circuit does not appreciably improve communications.

 

I'm rather curious what vintage Accesspoints you have installed. If you're into hypothesis testing, you could try "breaking" your system by removing the Accesspoints and performing the RF bridging with DB-LL's. If my theory is correct, the DB-LL should refuse to RF couple the time shifted I1 communications.

 

I'm sorry if this has forced you to incur the expense of the HL package, but at the same time, I'll welcome another set of eyes on the problem. HL2 and the accompanying PLM is the only means that I know of to disable PLM retries. This, and the ability to control Hop counts, is what led me to this problem.

 

IM

[IndyMike]

Hello Illusion,

 

I'm glad to hear that your replacement of the KPL's continues to be a good fix. I also apologize if I am dragging you back to the Dark Side...

 

Per my note to ergodic above, please be mindful of how you are performing the RF bridge to the isolated PLM. My test above indicates that "older" Accesspoints are more tolerant of the I1 communication timing. I can't quantify "older" - my accesspoints are V1.0 with no date code.

 

IM

[Illusion]

Copy,

 

Initially I plan on just isolating the PLM. As it is dual-band that should be the RF coupling right there, right?

 

I have lots of old Access Points to play with as well. So if I need to add an AP to the filtered side with the PLM I can do that as well.

[IndyMike]

Illusion,

 

Correct - the Dual band PLM should RF couple itself. My comment was aimed at whatever device you use for RF coupling on the "House" side of the filter. The Accesspoints (Old) appear to perform better.

 

I've been trying to verify that the AP's perform better on the house side - unfortunately today is one of those "good" communication days. I haven't seen a failure since 5 this morning. Apparently the Dark Side took the day off.

[Illusion]

My comment was aimed at whatever device you use for RF coupling on the "House" side of the filter. The Accesspoints (Old) appear to perform better.

Oh, of course! I was not even thinking of the house side. I have multiple dual-band switches, Lamp-Lincs, and Access Points, so coupling to the house side was not in my thought process. The purchase of the dual-band KPLs, and SwitchLincs was largely because of my nightmarish KPL investigation. I figured if I appeased the Insteon Dark Force gods with a gift of additional revenue maybe they would look favorably on me.

 

It is funny, at the point I ponied up for Dual-Band switches and KPLs recently I had subscribed so strongly to the IDF belief system that I did not know whether they would make my system better, worse, or the same. I just figured, what the heck, here is something I haven't tried yet!

 

Right now I have 6 Access points trying to appease the IDF. It sorta seems to be working. I have some distant i1 devices that are finicky, but cannot bring myself to full investigation. Right now good enough is good enough. Maybe if the girl is gone I can investigate this phenomenon with those troublesome devices, but the significant neighbor will not be interested in more Insteon testing annoyances at this point. I will bide my time.

 

When I re-institute the i1 KPLs good enough will not be good enough.

 

Great thread, BTW, IM.

If I can verify this issue with only i1 devices, then at least $$ can fix it.

[tedbusen]

I am not nearly as sophisticated as you guys on this topic. However, what you all have discussed here I have been able to prove buy throwing $$ at it (No pun intended). The older devices don't work well with the new I2 devices especially the older KeypadLinc's. 3 Months ago I upgraded to the new dual band PLM, and my troubles began. My system has been in place for over 4 years and although not perfect was better than what is was after the new Dual Bank PLM.

My configuration is 169 devices (includes the counts for the individual buttons on the 8 button KPLs), 4 Dual Band Access Points and the new dual band PLM. There are 401 links in the PLM. I am on firmware version 3.3.10.

I have moved the PLM and access points all over the house trying to eliminate communications issues. I have dedicated circuits. I even created two power strips with no filters in them and plugged all my devices into the same outlet as the PLM. Anything not up to v41 was a crap shoot. Sometimes they would link correctly building complex scenes and sometimes they would not. Watching at level 3 diagnostics kept showing 0 hops on various devices even though they all were within 5 feet on the same outlet as the PLM. I have filterlincs all over the house and everything else was turned off.

So, I suspected the PLM. Put the older PLM in the circuit and had the same results. At this point I did have a couple of v41 KPL's so I removed everything else but those and a couple of SL's and SLD's. Things worked fine. Put some v41 dual band LL's into the setup and they worked fine. Put some older version LL's and they were not reliable.

I would do a factory reset and restore on all the devices, rebuild the scenes and still have issues. I would look at the Device Links table and they would be corrupted. Either having duplicates or missing records. Then I tried the older version KPL's and had worst results. So.... decision time. Spend money or junk the system.

 

I decided to spend money. Upgraded all KPL's to V41. Interesting thing happened in this process. Roughly 2 out of 10 of the new devices from Smarthome were defective. Never had that problem before. Smarthome reps didn't believe me and questioned me thoroughly about the problems. KPL's had LED's that didn't work. Filterlincs had no power coming out of them. Two KPL's would not load completed manually or via ISY. One new LL didn't work at all. Of course this happened over a two month period since the bank account wouldn't support a one time purchase. Roughly $2K at this point. Yes, I have more money than sense. I have proved that many a time. Considering the purchases over time I have roughly $8k in this Insteon setup. And now with Smarthome reliability in question, I am wondering if I am throwing good money down a rat hole. If it wasn't for the ISY/UDI support, I would be out of here.

 

I have a total of 16 KPL's all v41 now, 14 LL's v.41, 13 SLD v.38, 6 SLR v.2c, 4 Dual Band OutletLinc Dimmers, v.40 and 1 DB SL v.42.

Not cheap.

Bottom line, now I can setup everything just fine. I very seldom see 0 hops left on configuration and setup even when installed in the various locations throughout the house. The old SLR v.2c's do give me problems setting up, but once I get them configured they also work.

My system works 99.9% at the moment. The system responds immediately, results are predictable and constant. My device links are correct and match ISY. The PLM table is the same as ISY.

 

I am interested to see if others agree with this assessment or was/is there a simpler solution?

[ergodic]

My general observation is the same as yours - the newer devices are more comm-reliable overall.

 

That is especially true with the PLMs IMHO. My V.91 PLM "fixed" a number of random, nagging (and completely illogical) "comm" problems.

 

The other thing I did was to put the PLM behind a filterlinc and force it to use RF-only. That made a big difference in reliability. Like you, I'm now at 99%+ and really quite happy with the system.

 

I've reached the conclusion that this improvement is somewhat due to better hardware in the devices. But more importantly - improved firmware.

 

I used to "buy ahead" to have spare devices handy. I no longer do that. I get what I need when I need it and I know its the latest. I will say SH has been good about replacing things when asked. But with 2-month old KPL devices that shouldn't even be a question since the warranty should have covered it.

 

I have mostly V36 (I think) KPLs and have no problems with them.

 

Get a glass of wine and take a look at a C1 or Crestron system cost -- you'll really feel much, much better. And if the problems you had started to show up for me, the first thing I'd do is exactly what you did (in fact I sort of did a couple of years ago).

 

Though from experience the first thing I'd do before any major rework is to replace the PLM with the latest rev and see what that did before doing anything else.

 

And that's my general advice to anyone trying to debug their system and getting more and more puzzled: Most Insteon problem are of the signal-suck/noise variety. But those are usually pretty easy to establish if not necessarily pin-down. But there also are these "X-factor" problems beyond that like you had. And if you arrived at that point, replace your PLM with the latest and put it behind a filter and force it to communicate with the APs exclusively over RF. That is my recommended topology - I haven't had any feedback that it ever made anything worse and it often improves things a lot.

 

I have not seen the problems you've had with newer SH device reliability. So I can't speak to that. Honestly I haven't really had a DOA unit from SH in a very long time and it's been quite a while since I've even had a failure (other than the older and terminally crappy PLMs.) Maybe just the luck of the draw. Dunno. I don't mess with success - there's only one direction it can go....

[tedbusen]

Thanks for the reply. I do feel better that you also have taken a similar approach to this issue.

I am now sitting back with my favorite wine, with my beloved wife, and watching TV with my Watch TV scene.

All is good