Duplicate Pushes

[macjeff]

I filed a ticket but was told to post here.   Its not all but many programs.

Pushes send over and over from UD Mobile.  One time for hours and I got like 3-4 per minute.

And ALL the devices get the pushes not just one.

They stop on their own.  And I can manually run the THEN of a program and it starts the loop again

But I have counter in programs and it only runs once.  Plus another parts of program only run once (As log shows) so its not the program

I did get somewhere today on this.  All I did is click on the notification server and reselect ALL devices and save it and so far so good.

So UD said to post the program.  Here it is with the PUSH line selected

***  So now I have reselected every PUSH line in programs that are sending duplicate programs, clicked update and save

So far now they are all working fine but we shall see as it comes in waves.

[DennisC]

To post a program, right click on the program name and select "copy to clipboard" all the way at the bottom. Then post the result here. It makes it much easer to read.

You should also post related programs (Driveway Alexa Trigger).

Could you tell us a bit about your system? What device are you using, what firmware & UI version? You mention UD Mobile, are you using Notification or node server, if so, on PG3, which version? Where is "Virtual Button" coming from - node server?

After receiving many notifications, right click on the program and select status. Note the Activity, Status, Last Run Time, Finished Time.

[macjeff]

Using ISY on Polisy.  5.5.9.  

Using Notification Nodeserver 3.4.6

Virtual Button is a nodeserver that you push a button to trigger a routine on Alexa.  Its easier than setting up variables to trigger alexa.  One thing to note- the button was only pushed once so the program only ran once.

Some programs dont have virtual button.  And one program is just a WAIT and then the notification server line.

The only common denominator is notification nodeserver that are in all the programs.

The last run time was right when the issue started but it duplicated for several minutes and all programs had been finished.

I added a variable counter to each program.  It reset at midnight last night.  After I got 8 of the notifications the counter said 1.  I then ran the program manually and it said 2.   Then I got many more so I checked and it still said 2. 

This could be apple issue.  But all of our devices get the same push randomly from almost every program but once the issue stops they all stop.  It last happened last Monday Morning.  Then again today so over a week later.

So I wont post the program unless you really want to see it as the counter, the last finish time, and other things prove the program has only been run one time but the notification itself gets stuck in a loop and keeps sending.

UPDATE-  On the ones that were doing it this AM, I just went to the notification server line of the program. I re-selected the device (ALL DEVICES) and then hit update.  I then saved the program.  So far they have not done it, but whatever is causing this could have been fixed on the cloud side (Apple or UD) so my fix may have just been a coincidence.

 

[MrBill]

To be honest I have no idea what you're doing or trying to do with that program.  You set 3 variables, wait one second, and set them again to the exact same values.  If all your programming is like that, no wonder you're having issues....

If the problem is the Notification nodesever repeating notifications, we and/or @Jimbo.Automates will need to look at the node server log.

In addition to @DennisC's suggestion to paste programs as text and post all related programs, you should also name your variables so that anyone can tell at a glance if the variable is Integer or State (or possibly Constant).  The general nomenclature that most use is the just make the first character of the variable name either s, i, or c, and then either add a period or underscore or simply capitalize the 2nd character....  c is a constant or a variable that never changes.  Some Examples of constants: cTrue = 1, cFalse = 0, cBlue = 255 etc  Using these makes programs more readable especially when you post them for someone else....

If
      $sAway = $cTrue
then

 

When $sDriveway_alert is to 1 it may trigger another program to run, when $iDriveway_alert is set it won't trigger another program, but might filter another program from running.

[macjeff]

I was fooling around with program and had copied some stuff and forgot to remove it.  The duplicate variable set is gone but that was after the issue.
 

the program has nothing to do with issue. I think it’s on apples side.  The push notifications get in a loop and keep coming in and then randomly stop.  And then days later start again. 
 

it only started about 10 days ago and random on programs. One day one program might do it and the next time another one and sometimes if you at the same time. I even did some test programs with just a push in the program and it did it eventually  

There’s nothing happening on the ISY side so it’s something to do with the push notification server or Apple   

[MrBill]

4 minutes ago, macjeff said:

the program has nothing to do with issue. I think it’s on apples side.  The push notifications get in a loop and keep coming in and then randomly stop.  And then days later start again. 

So your saying all apple push notifications are repeating?  yes that would be an apple issue.  If that's not what you're saying and its only those that originate in IoX and follow the path Iox -> node server -> Pushover -> apple server, then there's several possible points of failure before reaching apple.

If you think it's an apple issue, then why did you open a UD ticket? wouldn't an Apple case be more appropriate?

7 minutes ago, macjeff said:

And then days later start again. 

So the same notification randomly repeats itself without being re-triggered?  Or when it's retriggered it sometime repeats for awhile?  Again here you need to look at the node server log... the node server is designed to RETRY if it doesn't recieve a "200 Ok" response (or certain permanent failure responses).  Are the repeats in fact coming because a 200 is not being received?  it could even be your internet connection at fault.

Your wasting everyones time if your unwilling to check the nodeserver log, and show all related programs. (and if any variables are used in IF statement tell us if they are State Variables or Integer Variables.

[Javi]

Hi @macjeff,

 

23 hours ago, macjeff said:

They stop on their own.  And I can manually run the THEN of a program and it starts the loop again

The timestamps on your notifications are different so they are unique not duplicated by the app or by Portal.  There is the possibility that the Notification Node Server is looping, but if this was the case it would likely happen for all notifications not just a few programs which command the Notification Node Server.  You can check the NS logs to see if this is occuring. 

You state that when you run the program the loop starts. This indicates  you have something causing a loop somewhere.  The same command can be sent from the Node Server itself outside of programs.  My guess is that if you send the  notification directly from the Node Server, not from a program,  there will be no loop.  This would eliminate the Node Server as the issue and leave us with issues in programs.

[DennisC]

As stated by others, you are limiting the amount of assistance we can offer. As @MrBill indicates, knowing what the variables are doing, how the programs are written, and what might have been captured by the logs is important.

With so little information, I am guessing here, but the Pushover node server allows you to set the number of retries and an expiration field, what do you have these set at?

[macjeff]

2 hours ago, Javi said:

Hi @macjeff,

 

The timestamps on your notifications are different so they are unique not duplicated by the app or by Portal.  There is the possibility that the Notification Node Server is looping, but if this was the case it would likely happen for all notifications not just a few programs which command the Notification Node Server.  You can check the NS logs to see if this is occuring. 

You state that when you run the program the loop starts. This indicates  you have something causing a loop somewhere.  The same command can be sent from the Node Server itself outside of programs.  My guess is that if you send the  notification directly from the Node Server, not from a program,  there will be no loop.  This would eliminate the Node Server as the issue and leave us with issues in programs.

Ok here is all I know

• Happened 3 times so far in last 10 days   All three in the AM but not the exact time.
• When it happens ALL of the programs that are triggered during the time the problem starts send duplicate notifications.  
• Even a program that has one line to send a notification triggered once and then I got about 3-4 pushes a minute for 20-30 min the other day
• The program itself that sends the notification is only running once.  I can confirm that with a variable counter and other things that would be fired by the program and are not.  
• This usually goes on for about 30 min or less and then stops.  I will get the notifications about every 20 seconds. 
• If I check the program logs the last run and finish time is during those 30 minutes but lots of pushes since it last showed finished.
• It did not happen today.  Last time was yesterday.  And it happened on different days of the week
• If the notification node server is set to send to ALL devices (not just one) then all of them get all the duplicate pushes.

I don't need any assistance as its not on my end.  I agree with JAVI that its most likely a loop either happening in the Push Server or Apple.  Just wanted to report it here

[MrBill]

19 minutes ago, macjeff said:

I don't need any assistance as its not on my end.  I agree with JAVI that its most likely a loop either happening in the Push Server or Apple.  Just wanted to report it here

yes it likely is on your end, either an error in your programs, or very remotely possible a very bad internet connection at your location. 

The common denominator is you.   Multiple people using the node server, no one else is reporting this issue.  Many more people use pushover, doesn't appear at least from the users here that his is a problem either, doubt it's an apple problem either, we would hear about it somewhere else.  That leads us back to either the node server is repeating it, or your programs are creating it. 

BTW, FWIW... if you agree with Javi then your agree the problem is within your programs.  Read the last two sentences of his post again.

[Javi]

Hi @macjeff,

I think you misunderstood my post.  The likely cause IS in your programs. Less likely, but still possible, is an issue with the Notification Node Server.

We have no other user reporting duplicate user generated messages.  The issue is NOT with Portal as the timestamps when Portal received the requests to send  notifications are different.  The issue is NOT with Apple Push Service as portal IS sending these messages.

[macjeff]

1 hour ago, MrBill said:

yes it likely is on your end, either an error in your programs, or very remotely possible a very bad internet connection at your location. 

The common denominator is you.   Multiple people using the node server, no one else is reporting this issue.  Many more people use pushover, doesn't appear at least from the users here that his is a problem either, doubt it's an apple problem either, we would hear about it somewhere else.  That leads us back to either the node server is repeating it, or your programs are creating it. 

BTW, FWIW... if you agree with Javi then your agree the problem is within your programs.  Read the last two sentences of his post again.

I feel like I am being attacked here for posting.  I have been doing this for many many years.  I had the original ISY, and then the 994i, then Polisy, now IOP.  So I know this stuff pretty well, and the pushes have been in place since the notification nodeserver started supporting push (and not just PUSHOVER).  Never had issue until about 10 days ago.

With that said, I have had a busy week but forgot to post something very important. 

Last week I put ALL PROGRAMS into a folder and disabled it.  No programs could run.

I was out of town so really did not care.  I then rebooted my ISY.

After the reboot I got duplicate down and up PUSH notifications from the portal over and over for 30 min.  I have TWO ISY's and I rebooted them both at the same time.  I then verified both showed UP SINCE...... at the same time in the portal and they did and I continued to get duplicate push notifications from the portal.

Then they just stopped on their own without me doing anything.  I only get them from one of my ISY's because only one is registered with UD Mobile.

So my programs were disabled and the push came from the portal not my ISY. 

I was told to post the latest program by Javi so I did.  That was a mistake because unless there are two issues, it has nothing to do.  And the pushes came from all different programs.  I think my post led people to believe it was the program.

Any pushes during the period of duplication (Every one had PORTAL messages mixed in) would duplicate and then it would stop.

BTW...I then turned programs back on last week and no issues for the weekend but then got it again yesterday.

Maybe this is as simple as some sort of connectivity issue to the portal, but why would the portal say online for 2 days.  Like it happened yesterday (Wednesday) and right now it says online since Tuesday but I got portal down messages yesterday.

A screenshot below shows the portal messages. I had pages of them and again the portal showed it had been online for 20-30 min and I was still getting up/down/up/down.

Maybe I should just hide this entire conversation since it seems to be causing me more of a headache then the original issue!!!

 

I was just trying to see if anyone else had this issue. So far no issues today at all.

[macjeff]

5 minutes ago, Javi said:

Hi @macjeff,

I think you misunderstood my post.  The likely cause IS in your programs. Less likely, but still possible, is an issue with the Notification Node Server.

We have no other user reporting duplicate user generated messages.  The issue is NOT with Portal as the timestamps when Portal received the requests to send  notifications are different.  The issue is NOT with Apple Push Service as portal IS sending these messages.

see post I just sent.  I am going to back out of my own conversation as everyone is attacking me and thats not what the forum is supposed to be about.  I have helped many here.  I am very experienced with the ISY and you said to post so I did.

I just dont like responses like I am the problem.  It could be random internet drops, duplicated UUID on push server.  Lots of things.  But its not the programs.  Even disabled the problem happened with pushes from the portal.

[Javi]

Hi @macjeff,

Nobody here is attacking you, we are  simply trying to debug.  I have also authored many bugs, finding and fixing them is what makes you "good".

We are saying the likely issue is your programs or the Notification Node Server.  We have already narrowed down the possible causes but we cannot go any further without  the logs from the Notification Node Server when this occurs and/or running the tests I suggested.

IoX program execution is not linear.  A program can stop executing if the conditions (IF) are re-evaluated.  So we need to know if the Notification Node Server is receiving a new command for each notification in the loop or if the Notification node server is not getting a response from Portal and attempts re-sending the notification.

[macjeff]

But some of the duplicated notifications were not from my ISY at all.  They were from the portal saying Up or Down and that kept duplicating.

Doesnt that rule out programs?

And as I said either in email to you or here (getting confused) that I noticed now that every time I get these strange pushes from multiple programs all at the same time there are UP DOWN messaages also duplicated.  So I wonder if something is happening in connection.

I had up and down portal messages YESTERDAY and look at the timestamps on the portal.  It shows it was up yesterday.

See screenshots.  The difference in the two dates was a manual reboot but the important thing is that i got 20-30 up down duplicated messages from the portal at the same time as the driveway and others yesterday and portal shows UP!!!

If it said Wednesday it would correspond with the ISY offline stuff.  (please note I deleted my pushes from yesterday because I had over 100 so I cant screenshot them.

Lets just take a wait and see approach.  It could be portal maintenance or some other glitch and it may go away on its own.  Also the periods are under 30 min where push notifications duplicate and I get the up down portal messages.

But PLEASE dont waste your time looking at logs just yet!!!

[kclenden]

1 hour ago, macjeff said:

I feel like I am being attacked here for posting.

I know it feels that way, but you just have to flip things around to understand what's going on.  You presented a mystery in a place populated by very logical thinking people who enjoy solving puzzles.  While they want to help you, they also want to solve the mystery.  So they asked for additional information but for one reason, or another, you haven't provided it.  So they feel frustrated and it comes out feeling like an attack.

[macjeff]

55 minutes ago, kclenden said:

I know it feels that way, but you just have to flip things around to understand what's going on.  You presented a mystery in a place populated by very logical thinking people who enjoy solving puzzles.  While they want to help you, they also want to solve the mystery.  So they asked for additional information but for one reason, or another, you haven't provided it.  So they feel frustrated and it comes out feeling like an attack.

I appreciate your response.  Been a long day and glancing at the messages just got me stressed

We think we know the issue.  The timeout on the pushes needs to be increased.  Logs show them timing out at 10 seconds so it tries again but it already was received.  It just timed out before it got the success message.

Will post once we know for sure.