ZIGBEE Smart Meter Module Down?

[LFMc]

This morning at 5:45am, my rest communications to my smartmeter went into error mode and then stopped even trying to communicate 15 minutes later. Now looking at the rest logs, it only shows an entry when I restart the ISY994i saying "system start".  I moved the ISY to within 3' of the meter and still nothing. On the ISY Dashboard, My Electricity module show the time as "unsynchronized".  

There is an LED at the base of the Zigbee antenna that is not on. It faces away from me in my cabinet, so I am not sure what it's normal illumination should be. Any ideas? 

I have not re-seated the Zigbee module in the ISY yet. Does this seem to be a dead zigbee module? 

Any help would be appreciated.

[LFMc]

Just an update. Hopefully this helps someone else out in the future.

Thanks to UDI's assistance, @Michel Kohanim, we pretty much ruled out any of my 2 disconnected Zigbee items having a problem, so that left my Oncor smart meter as the culprit. 

After 4-5 phone calls over 3 weeks to Oncor, they **finally** conceded that I knew what I was talking about and agreed to pay a visit at my site to test the Zigbee HAN system on the meter as they claim they could not test the HAN from remote. I decided to put both my devices (SmartenIT repeater and ISY) back in to search mode and did the same on the Oncor website, smartmetertexas.com on 8/28 which puts the meter, supposedly, into accept devices mode. I did this because Oncor apparently will do anything they can not to have any contact with a customer. From experience, I suspected they would not honor their promise to contact me before and during the visit so I would know what is going on and be able to test the system.

Just as I suspected, 3 days later, my Smart Meter's Zigbee network just woke up and my 2 smart home Zigbee devices magically reconnected to the Home Area Network on their own. No contact before or after on what Oncor did to fix their meter problem. It now appears they issued a remote reset on the meter as at the same time I had 3 PCs shut down and reboot. But apparently it was quick enough of an outage that my ISY did not reboot and it isn't on a UPS. 

So now when I go to SmartMeterTexas.com to see the status of my devices that should be connected to Oncor, it says the meter is still waiting for me to connect my devices. This is very weird because both my Zigbee devices are showing on their side that they are connected, but the Oncor side is showing they have never authenticated. 

Sounds like more phone calls are needed to get their software working correctly. In the end it just seems like big companies like this will typically do most anything to not expose problems on their side. 

So, I should have insisted originally that they reset the meter from remote. I did ask originally for them to do this and they told me it wasn't possible. They may have been referring to just a Zigbee/HAN reboot only, but I would have taken anything to get the meter's network back up. 

[Michel Kohanim]

Hi Leon,

Thanks so very much for the update. About a year ago, there were rumors that TX (and Oncor) were removing all funding/resources for the HAN side of the Smart Meter. Perhaps the rumors are true?

With kind regards,
Michel

[LFMc]

1 hour ago, Michel Kohanim said:

Thanks so very much for the update. About a year ago, there were rumors that TX (and Oncor) were removing all funding/resources for the HAN side of the Smart Meter. Perhaps the rumors are true?

Being there is a PUC rule in place (§25.130(j)(8)) requiring HANs connections on the meters, I would have a hard time believing they could justify taking it away. In fact, I believe they are due for a major update to the smartmetertexas.com web site that should include more features such as "green button connect my data" universal access, but I cannot be sure. I found this web site that seems to be an advocate for more consumer access in all states. http://www.missiondata.io/activities#index  They specifically discuss activities state by state. There doesn't appear to be much recent activity in Texas. But there is an un-referenced comment stating: "Third party access has been challenging due to poor technical support of SMT" on the Texas activity.  I can personally testify as to the level of difficulty they made it for me to get someone who even knew what I as talking about to address my problem.

What I had to do was to ask the Oncor gate keeper if they were familiar with the PUC ruling §25.130(j)(8) (pg. 69 of this document http://www.puc.texas.gov/agency/rulesnlaws/subrules/electric/25.121/31418adt.pdf) and were they taking responsibility for not complying (after I got their name of course). That immediately got me to a supervisor in charge of dealing with HAN device  issues.  Then I was told it would take up to 10 working days to get around to it. In reality, it took about 5. 

Clearly there seems to be a low priority and a lot of foot dragging about getting anything done with valid HAN issues. On the other hand, I can also see their side as they probably get LOTS of issues with people buying HAN devices and can't figure out why they just don't work by just turning them on. 

Please feel free to update me if you get more detailed information. I would think that it might help UDI to be a part of the Mission:Data alliance. 

--Leon

[Michel Kohanim]

Hi Leon,

I will definitely keep you posted. As far as I remember, Oncor filed an advice letter with PUC to get rid of having to support the HAN side of the smart meters. 

With kind regards,
Michel

[LFMc]

5 hours ago, Michel Kohanim said:

I will definitely keep you posted. As far as I remember, Oncor filed an advice letter with PUC to get rid of having to support the HAN side of the smart meters. 

Thanks Michel,

Their support for a HAN user is fairly non-existent as it is. 

Just so you are aware of some other changes that have occurred since Oncor rebooted my meter.  I had my SmartenIT repeater still located next to my smart meter instead of mid-distance between the meter and the ISY.  But since the meter reboot the ISY has been almost flawless on data flow (from the meter) which has never happened before without the repeater and other enhancements strategically placed.  So I turned off the repeater and the data flow is still near perfect at the 80+ feet through the house from the meter, even without the aluminum reflector.

I bring this up because of the prior range issues I had trying to get the Zigbee interface on the ISY to connect up to the meter. I know it caused you (and me) a lot of concern about why the Zigbee range was so limited. Now it seems the range has easily more than doubled since the reboot. I have no idea if the Oncor tech somehow boosted the meter's Zigbee power output or if it was just the reboot of the meter that solved the "no connection" and range issue at the same time. 

Clearly the option of getting the TDU to reboot the smart meter is a valid option when there are range or connectivity issues.  This is also a confirmation of the ISY's Zigbee interface as being solid and more than adequate for a much longer range (as the UDI specs state), even through multiple walls.

I'll update you if I see any further anomalies in my SEP communications. 

[Michel Kohanim]

Hi Leon,

This is wonderful news albeit quite strange. Thanks so very much for the update!

With kind regards,
Michel

[petteyg359]

FWIW, Oncor has quality control issues with the meters. At three different addresses, a new HAN device worked for several months and then suddenly the stopped, and immediately started working again after I called Oncor and demanded they reset the Zigbee device in the meter itself. Maybe they're doing firmware updates and they don't bother to check if their updates continue to work with HAN devices, or fail to reset some state they require to connect or something. Anyway, if it ever stops working, assume Oncor broke the meter again before assuming your device is actually faulty.

[LFMc]

1 minute ago, petteyg359 said:

FWIW, Oncor has quality control issues with the meters. At three different addresses, a new HAN device worked for several months and then suddenly the stopped, and immediately started working again after I called Oncor and demanded they reset the Zigbee device in the meter itself. Maybe they're doing firmware updates and they don't bother to check if their updates continue to work with HAN devices, or fail to reset some state they require to connect or something. Anyway, if it ever stops working, assume Oncor broke the meter again before assuming your device is actually faulty.

Thanks for the corroboration of Oncor meters being faulty. One very weird situation I haven't mentioned yet was after they reset my meter (without telling me) is both of my HAN devices reconnected automatically without permission. I had previously removed both of them from my HAN list of approved devices. I verified this through smartmetertexas.com. So they both "logged in" to the Zigbee HAN network and were working correctly without showing at the SMT dashboard or HAN log. I even let them run for a week or so in that condition to see what would happen, i.e. would Oncor wake up and figure out something was wrong.

What this leads me to believe is possibly Oncor has an "accept all" setting without credentials on the HAN in an effort to appease my haranguing (albeit nicely) them over their meter causing the problem. I finally did put in a request via the SMT website to officially register my devices just in case Oncor came back and turned on security restrictions, if my theory is correct. 

Lastly, if you are registered on SmartMeterTexas.com, you can see the last firmware update date on the site under the meter information. That was one of the first things I suspected, but my meter, at that time, had not been updated in a few months, but today I see that they updated the meter about two weeks of rebooting it. Here is where you can see the info on the meter.

Thanks again for the feedback. 

[Michel Kohanim]

Thank you both for the details. The fact that ISY automatically rejoined means that the only thing they might have done, was a reboot. If they factory reset the dongle, then you would have had to go through the whole registration process again. So, I suspect some of these Zigbee modules may require a reboot after a while.

With kind regards,
Michel

[LFMc]

On 9/4/2019 at 9:02 AM, Michel Kohanim said:

I will definitely keep you posted. As far as I remember, Oncor filed an advice letter with PUC to get rid of having to support the HAN side of the smart meters. 

BTW, I got this email from SmartMeterTexas.com yesterday. It appears that Oncor may be expanding their offerings to us involved in meter watching.  With them being able to provide data via an API, would this mean we can (possibly) dump Zigbee and consume the data directly over the internet via a (future) nodeserver?  Probably just a hypothetical question at this point.  But reading between the lines, this may be an attempt to dump dealing with Zigbee in the future (as your previous message alluded to).  Right now I am getting data every 30 seconds and have the option of polling more often. I bet that any API they allow to see their data directly will never allow polling that often.  I will have to do some more research on what the details are in this "approval" from the PUC involves, after all,  "the devil is in the details".  There are lots of very vague claims in the email. 

Another interesting issue occurred in August. Some Oncor users are buying power through brokerage wholesalers who pass along "the cost" of the electricity for a flat fee of $10 per month. What they don't tell you is they are buying power hourly on the spot market. Last August during a particularly hot day, the spot market for power in Texas jumped to $9.10 per KWHr for an hour in the peak power hour of the day. My friend uses one of these brokers called "Griddy" and his bill for electricity was $135, for one hour!  Ouch! I did't pay that much for the entire month. 

Email from SMT: 

"Due to recent approval by the Public Utility Commission of Texas, Smart Meter Texas will be releasing new functionality on December 10, 2019. A few key highlights of improvements and changes are the following:

• Ability to receive your data via an Application Programming Interface (API)
• Ability to receive your energy data via email
• Enhanced mobile awareness
• Enhanced to display 24 months of data
• Improved access to generation information
• An improved customer experience with a new website look and feel

Watch for future emails for additional information and action required.

 Thank you for being a Smart Meter Texas user.

Regards,
Smart Meter Texas Team"
 

[LFMc]

Latest on SMT 2.0 which is rolling out soon. 

https://tcaptx.com/legislative-watch/blog-puc-green-lights-smart-meter-texas-2-0-expected-in-2020

 

The one item that is of close interest to me is the removal of using the SMT web site to gain HAN permission and attachment.  Looks like the ball is being handed off to the local TDU. 

"Smart Meter Texas 2.0 will no longer support Home Area Network functionality, although individual transmission and distribution utilities must maintain access for existing customers who request it. Home Area Network devices include smart thermostats and smart appliances that work with digital smart meters."

[Michel Kohanim]

Hi Leon,

Ah, finally. That's what I was looking for! There are two things of note:

1. SMT 2.0 -- it seems that they are disconnecting Smart Meters from the Home Area network. This means that your ISY will no longer connect to the meter (it's like a smart appliance)
2. The email with regard to access to data, please note that I suspect this is Green Button data which is ALWAYS 24 hours behind and, at best, 5 minute interval (usually 15 minutes or longer)

So, the first thing I suggest you do is to see whether or not I am wrong with regard to #2 and Green Button. If I am wrong, then there's still hope to create a node server that connects to their APIs in the cloud. With respect to #1, I guess they will have to keep you connected if you request it.

With kind regards,
Michel

[LFMc]

3 minutes ago, Michel Kohanim said:

Hi Leon,

Ah, finally. That's what I was looking for! There are two things of note:

1. SMT 2.0 -- it seems that they are disconnecting Smart Meters from the Home Area network. This means that your ISY will no longer connect to the meter (it's like a smart appliance)
2. The email with regard to access to data, please note that I suspect this is Green Button data which is ALWAYS 24 hours behind and, at best, 5 minute interval (usually 15 minutes or longer)

So, the first thing I suggest you do is to see whether or not I am wrong with regard to #2 and Green Button. If I am wrong, then there's still hope to create a node server that connects to their APIs in the cloud. With respect to #1, I guess they will have to keep you connected if you request it.

With kind regards,
Michel

Yes, I think we are reading it the same way. They will offer Green Button data access so users do not have to create an SMT account. Also, smartmetertexas.com (which technically is for all TDU's customers if the TDU subscribes to it) is not going to handle HAN provisioning and the TDU has to do it themselves. I can't see them eliminating access in any way for HAN customers as it would cause so many devices to no longer work. Hopefully they are not taking away anything, just rerouting access permissions. 

Yes on the 24 hour delay also. Which really works out to "up to" 48 hours of delay as it has a 24 hour wait that starts from the last day's end. Hopefully this will not be the case going forward.

[Michel Kohanim]

Hi Leon,

Agreed. It's quite a shame since, in the 21st century and moving toward more dynamic pricing structure, having access to real time energy information is a must. I suspect they just wanted to get rid of thermostats and appliances. Would you mind asking them whether or not they would also discontinue supporting IHDs (In Home Display) as it was not specifically spelled out. ISY is pretty much an IHD from the Smart Meter perspective.

With kind regards,
Michel

[LFMc]

5 hours ago, Michel Kohanim said:

I suspect they just wanted to get rid of thermostats and appliances

Michel,  I seem to think your interpretation of the SMT 2.0 notice shows some concern about participation of the ISY going forward. I did do some more research and I do not read into it any reduction in the HAN connection services based on device type. It seems the vast majority of this change is centered around the Green Button initiative to reduce the effort needed by a consumer to get access to their data or to give 3rd party access to their data. The additional item in this change is moving the process of opening HAN permissions for the customer to add a device to the TDU. I think the statement about "existing" customers can be taken incorrectly, not as a grandfathered requirement but just as a current user permission requirement. I.E. you have to be an Oncor customer before they can let you add a device to the HAN.  I cannot find anything, at least yet, that shows any weakening or watering down of the requirement by the PUC in Texas that mandates allowing customers real time access to the meter data via the HAN or the optical plug in, regardless of the device type. 

Are you seeing or know of something else that concerns you in this area? I know you mentioned something previously in this thread. I'd be glad to further research it with some input/direction from you. 

[Michel Kohanim]

Hi Leon,

Yes, I am quite concerned because another utility here in CA is considering abandoning the HAN side of things. From my perspective, it would be a waste of billions of $. But, the counter argument is that the Smart Meters do still support remote meter reads by the utility and thus the investment in AMI (Advanced Metering Infrastructure) is not wasted.

I still do not see any value in Green Button except for historical validation and profiling services (which still need something to read real time energy info from the breaker).

With kind regards,
Michel

[LFMc]

On 10/10/2019 at 11:18 AM, Michel Kohanim said:

Hi Leon,

Yes, I am quite concerned because another utility here in CA is considering abandoning the HAN side of things. From my perspective, it would be a waste of billions of $. But, the counter argument is that the Smart Meters do still support remote meter reads by the utility and thus the investment in AMI (Advanced Metering Infrastructure) is not wasted.

I still do not see any value in Green Button except for historical validation and profiling services (which still need something to read real time energy info from the breaker).

With kind regards,
Michel

One year on, and Oncor is still sending my ZS data without any issues.  Have you heard anything further? 

[Michel Kohanim]

@LFMc,

I am so very glad to hear. No, I have not hear anything else.

With kind regards,
Michel

[Vaggeto]

Hello, such an interesting thread!

 

I purchased a Rainforest Eagle device back in 2015 and user SMT to get it provisioned and working properly, but there were a few instances where it stopped connecting to the meter and as you noted, they were quite lacking in their ability to troubleshoot and eventually I just gave up.

I have solar though, and over the years power service providers started new policies which limited access to good time of use plans such as free nights for those who wanted to get paid for the solar power they send back to the grid. So ideally I would prefer to know if I'm currently under-utilizing my my solar power generation and take advantage of that with live data.

I looked into the API connection that you discussed, and overall it seems beneficial but does have limits on queries allowing 2 reads per hour and no more than 24 in a day, but live readings are much more useful for me.

 

So I'm trying to get my HAN device working again and I have a couple questions.

1) Can you please confirm your device previously registered, still works?

2) Have you needed any support from oncor? I've just tried emailing Oncor support HANsupport@oncor.com to see if they can help and I was wondering if anyone has tried contacting them yet? Since my device hasn't connected successfully for likely 1.5 years, I'm afraid they will tell me I'm out of luck and they won't help.

3) Has anything changed in regards to this requirement that they support HAN devices that you mentioned in the PUC ruling? I was never aware of this but obviously their lack of support for new devices goes against this, so are they just openly not complying or did that change somehow in a future ruling?

 

Thanks so much, the discussion was extremely beneficial and is appreciated!

 

 

[LFMc]

13 hours ago, Vaggeto said:

So I'm trying to get my HAN device working again and I have a couple questions.

1) Can you please confirm your device previously registered, still works?

2) Have you needed any support from oncor? I've just tried emailing Oncor support HANsupport@oncor.com to see if they can help and I was wondering if anyone has tried contacting them yet? Since my device hasn't connected successfully for likely 1.5 years, I'm afraid they will tell me I'm out of luck and they won't help.

3) Has anything changed in regards to this requirement that they support HAN devices that you mentioned in the PUC ruling? I was never aware of this but obviously their lack of support for new devices goes against this, so are they just openly not complying or did that change somehow in a future ruling?

Hi V,  welcome to the confusing world of HAN devices. :-9

After Texas shut down HAN management via SmartMeterTexas.com, I have had no interaction with them. About 4 weeks ago, I somehow accidentally reset my ISY trying to copy the SD card in preparation of upgrading it to 5.x.  After restoring it, the ISY no longer talks to my meter.  My SEP repeater does still talk to my meter, so I probably need to ask Oncor to open up the "join" feature on the meter so my ISY can reconnect to it. 

So, you are ahead of me trying to get Oncor to respond. I have had actual phone discussions in the past with Oncor engineers and they did send a tech out to my house once to check the meter's HAN function. So they are responsive to me prior requests. It seems that once or twice a year my meter's HAN function locks up and stops feeding data to me. It turns out if power is removed from the meter (i.e. hard reset), then the HAN function returns to normal for another few months. So when that happens, I have to wait for a power outage or go through the trouble of convincing the Oncor service person that I know what I am talking about and cajole them into humoring me and resetting my meter. Of course that shuts down the power to the house for about 1 second and resets everything. ?  Simply rebooting my internal HAN devices never fixes the problem. 

I will probably call them this week and cross my fingers that I get someone that even knows what a HAN is.  Please let me know how you contact them and how it goes. If I succeed before you, I will post it also.

As to your last question, I have no idea if there has been any change releasing them from the PUC ruling that they have to support HAN devices. 

[LFMc]

Just Googled Oncor han support and saw this article. I pray it is not true. But who knows.

https://www.rainforestautomation.com/state-texas-residents/

 

[Michel Kohanim]

@LFMc,

CA is following through with the same unwise decision (personal opinion). Real time energy information is key for all permutations of Green. Now, something that didn't require any installation, requires CTs and professional installation in the breaker panels (CTs) or complete replacement of the breaker panels (Span.io). 

With kind regards,
Michel

[LFMc]

1 minute ago, Michel Kohanim said:

@LFMc,

CA is following through with the same unwise decision (personal opinion). Real time energy information is key for all permutations of Green. Now, something that didn't require any installation, requires CTs and professional installation in the breaker panels (CTs) or complete replacement of the breaker panels (Span.io). 

With kind regards,
Michel

I agree, obviously.  I think this is possibly a legal issue, at least in Texas. The consumers had smart meters forced on us in the name of "money saving" and ability to manage usage. In addition we have to all pay a fee every month for the benefit of having a smart meter. Given the onslaught of bad press this last week about people being billed $10,000 for power because they were on "wholesale" contracts, this refusal to support HAN devices will make them look even worse in the public's eyes. 

I guess now would be the time to bring this issue up with the media and with the legislature. 

[LFMc]

More depressing news. Some how the TDU's have convinced the Texas Legislature that HANs are not needed for "new" customers. Now you have to go the the SMT web site and click on the "green" button to get a "demand" reading. Totally useless and is nothing but a gimmick to appease some users. How would you ever know that your billing rate just peaked to 1000x higher for the next hour?  We are going backwards.

 https://www.utilitydive.com/news/texas-regulators-approve-new-smart-meter-rules-limiting-access-to-real-tim/576093/