apostolakisl Posted August 20, 2018 Author Posted August 20, 2018 4 hours ago, MWareman said: Not on my ISY... It sends the F or C as well.... I don't know how that is possible. There are no options that I am aware of to change that and mine is sending a number right now. It updates every time the number changes. I have a state variable for both humidity and temp that just updated with a 94 for temp and 43 for humidity at 8:31:37 (about 15 minutes ago). I have been running this network resource now through quite a few firmware versions, probably starting in the 4's through the present 5.0.13c. The only difference is that now I'm on the portal so I'm having to first have to loop it back via a LAN network resource to the originating ISY, then once I have it as a standard variable I can send it to the other ISY without exceeding the allowable number of characters. Quote
MWareman Posted August 20, 2018 Posted August 20, 2018 Well - I was testing with email substitutions. To be sure, I tested with network substitutions and got the same result.... Using this network resource: The response, in the Apache log on the receiving web server, was this: 192.168.1.20 - - [19/Aug/2018:21:47:01 -0500] "GET ?test=75 F" 400 0 "-" "-" The "<space>F" is most definitely there... (5.0.13C in my case - but this was the case as long as I have been doing substitutions) My only guess.... Could the ISY be truncating the incoming request at the un-escaped space - causing the request to succeed... and the Portal handling the request properly (failing it..)? Edit: @Michel - Seems we need a .RAW for all modules (including the Weather one) as a method to suppress the UOM being added to the output of the substitution... Quote
apostolakisl Posted August 20, 2018 Author Posted August 20, 2018 45 minutes ago, MWareman said: Well - I was testing with email substitutions. To be sure, I tested with network substitutions and got the same result.... Using this network resource: The response, in the Apache log on the receiving web server, was this: 192.168.1.20 - - [19/Aug/2018:21:47:01 -0500] "GET ?test=75 F" 400 0 "-" "-" The "<space>F" is most definitely there... (5.0.13C in my case - but this was the case as long as I have been doing substitutions) My only guess.... Could the ISY be truncating the incoming request at the un-escaped space - causing the request to succeed... and the Portal handling the request properly (failing it..)? Edit: @Michel - Seems we need a .RAW for all modules (including the Weather one) as a method to suppress the UOM being added to the output of the substitution... Unless humidity puts a % sign, then it is not that. Humidity also fails via the portal. I would not expect the portal to pay any attention to the content of the message but rather simply act as a forwarder. And humidity works fine direct from isy to itself or isy to another. Quote
MWareman Posted August 20, 2018 Posted August 20, 2018 Unless humidity puts a % sign, then it is not that. Humidity also fails via the portal. I would not expect the portal to pay any attention to the content of the message but rather simply act as a forwarder. And humidity works fine direct from isy to itself or isy to another. I’ll take a look at humidity tonight. I’d expect it to include the % - since UOM is almost always included in substitutions. The easies way to test is an email substitution sent to yourself.As far as Portal - no, it’s doing more than simply forwarding. If it were simply forwarding then the username and password you use would be that of the ISY.... Normally, the web server software http decodes incoming requests before passing them to the software for processing. Portal runs a full HTTP server process (something like Apache or Lighttpd, I don’t know for sure...) - and likely http decodes requests fully before passing them to the code that UDI has running there. On ISY, it’s an extremely light HTTP server process - and I’d guess that it’s not fully decoding the incoming requests - and instead they are getting truncated at (in this case) the space - allowing the requests to succeed to your ISY but fail via the Portal. Quote
apostolakisl Posted August 20, 2018 Author Posted August 20, 2018 16 hours ago, MWareman said: Not on my ISY... It sends the F or C as well.... A percent sign does show up in the email for humidity. None-the-less, it works. I tried test REST commands to my isy putting a %20 after a number (url for "space") thinking maybe isy truncates at a space, but it does not, it gives an error. For example, http://192.168.1.9/rest/vars/set/2/26/8%20 So there must be some other character there that tells it to truncate . . . cause it works when you use variable subsitution for temp and humidity. I would expect that the portal is not "understanding" the REST command, but rather just truncating all credentialing since it has already established a session with the target isy. There really would be no need to understand or interpret the command at the portal level. Quote
MWareman Posted August 20, 2018 Posted August 20, 2018 %20 is a properly encoded space though in your test - it will get interpreted as a literal string by the Portal and the ISY - and when you try to stuff a string into an integer it will fail. Basically, that’s what I believe is going on here.... Your test is basically causing the REST call to your ISY to fail in the same way the REST call to the portal is failing. When the space is inserted by the substitution, it will not be encoded at all... I think the Portal handles that - and then fails (because the value is not simply an integer) while the ISY is truncating at the space leaving only the integer causing the request to succeed. Quote
apostolakisl Posted August 20, 2018 Author Posted August 20, 2018 37 minutes ago, MWareman said: %20 is a properly encoded space though in your test - it will get interpreted as a literal string by the Portal and the ISY - and when you try to stuff a string into an integer it will fail. Basically, that’s what I believe is going on here.... Your test is basically causing the REST call to your ISY to fail in the same way the REST call to the portal is failing. When the space is inserted by the substitution, it will not be encoded at all... I think the Portal handles that - and then fails (because the value is not simply an integer) while the ISY is truncating at the space leaving only the integer causing the request to succeed. Except F and % sign apparently are part of ${mod.weather.temp.current} and ${mod.weather.humidity} YET, they still work direct ISY to ISY REST to a var. The exact same thing through the portal does not work. So I don't know what code is in there that tells the receiving ISY to ignore the string and just use the integer. And why this is lost with the portal? Whatever code it is, it is not %20. %20 seemed like a good candidate since in the email that seems to be the only thing separating the integer from the F or % sign. Quote
MWareman Posted August 20, 2018 Posted August 20, 2018 YET, they still work direct ISY to ISY REST to a var. The exact same thing through the portal does not work. So I don't know what code is in there that tells the receiving ISY to ignore the string and just use the integer. You are thinking this the wrong way around. The portal uses a very mature web server process - and it’s likely got code to handle unencoded spaces and pass them down to the portal code (causing it to see a string and an error when it tries to stuff the string into an integer variable). The ISY has an extremely lightweight web server process, and in all likelihood is simply truncating rather than take up space with the code to properly parse. This causes the remains to be an integer, and that fits the integer variable. The correct response is an error. You should not be able to send a string with a F or % to an integer variable. Ideally, the ISY should provide a substitution variable that suppresses the UOM - resulting in an integer that works in both ISY and Portal. This is already true for the .ST value (using .raw) - but does not currently work for weather module variables. The problem here is that the ISY accepts the request (it shouldn’t) - not that the Portal should (its behaving correctly and rejecting the non-integer attempt to set an Integer variable). The correct fix is the ability to .RAW the weather module variable substitutions. Until the ISY can support this, it must be done by an external script. Quote
apostolakisl Posted August 20, 2018 Author Posted August 20, 2018 43 minutes ago, MWareman said: The ISY has an extremely lightweight web server process, and in all likelihood is simply truncating rather than take up space with the code to properly parse. This causes the remains to be an integer, and that fits the integer variable. No, I tested this and posted it a couple posts ago. I just did simple REST commands ISY to ISY and added %20 followed by random letters to an otherwise functioning REST command. ISY returned error every time. ISY to ISY using the weather module variable substitution does not. Quote
MWareman Posted August 21, 2018 Posted August 21, 2018 1 hour ago, apostolakisl said: No, I tested this and posted it a couple posts ago. I just did simple REST commands ISY to ISY and added %20 followed by random letters to an otherwise functioning REST command. ISY returned error every time. ISY to ISY using the weather module variable substitution does not. I think you may be misreading me. By adding %20 - you are causing it to fail because the ISY sees the characters %20 (NOT a space). It's a string. So, it fails. By adding an actual space followed by any character (like a F or %) - the ISY is truncating at the space. This leaves the integer and the write to the variable succeeds. Trying with a browser will behave as if you manually tried %20 - since browsers will implicitly change spaces to %20's.. Your experience is exactly as I would expect if the issue is as I speculate. Only UDI can confirm (or refute!) for sure.. Quote
apostolakisl Posted August 21, 2018 Author Posted August 21, 2018 I tried two REST commands that set a variable, one ISY to ISY, one ISY to portal to ISY. First off, I just sent a single integer value to confirm all my settings, no problem on either setup. Second, I tried appending that same REST command with a space and letter. ISY to ISY it truncated off the letter and worked fine, ISY to portal to ISY gave error. In summary, it appears the portal does process the REST command and does not truncate and therefore fails. Seems like an unnecessary thing for the portal to do (process the REST command rather than just forward it on through). But my workaround does fix it (the workaround being to first network resource the weather module value back to yourself so that it becomes a normal variable) TAKE HOME POINT: The knowledge that you can network resource a weather module variable back to yourself and clear out the useless units and open it up for using as a comparative value or do math on it, or send it to another isy via the portal is worth something. To the best of my knowledge, looping a weather module value via a network resource is the only way to use a weather module value as something to compare to anything but a fixed value. For example, if inside temp (from a thermostat) is higher than outside temp (from weather module). Quote
MWareman Posted August 21, 2018 Posted August 21, 2018 1 hour ago, apostolakisl said: I tried two REST commands that set a variable, one ISY to ISY, one ISY to portal to ISY. First off, I just sent a single integer value to confirm all my settings, no problem on either setup. Second, I tried appending that same REST command with a space and letter. ISY to ISY it truncated off the letter and worked fine, ISY to portal to ISY gave error. In summary, it appears the portal does process the REST command and does not truncate and therefore fails. Seems like an unnecessary thing for the portal to do (process the REST command rather than just forward it on through). But my workaround does fix it (the workaround being to first network resource the weather module value back to yourself so that it becomes a normal variable) While it certainly would be nice is the Portal and ISY behaved the same - but it is correct what the Portal is doing and not correct what the ISY is doing. What you are trying to do should always have failed. The Portal is not 'processing the rest command' - nor is it 'not truncating'. Processing the full string is likely done by the Load Balancer (or other Amazon infrastructure) and is entirely correct. Truncating at the space is, per all standards, the incorrect thing (even though it's what allows your use case to succeed). It's a lack of processing code in the resource constrained ISY... I've said before.. adding the UOM to substitutions is the documented action. I've previously asked for a .RAW option on substitutions to direct the substitution to be made without the UOM added - but so far it's only applicable to the .ST value of nodes. If this concept were expanded to all substitutions that would solve this issue in a more sustainable way. If you have two ISYs - of course you could (as you document) send the values from one ISY to the other - relying on the truncation 'bug' to convert to an integer. Then you could send it back - or to Portal - now as an integer variable. I would caution that because you would be taking advantage of a likely unintentional 'feature' that it may stop working (or start functioning consistently) at some future version... Quote
apostolakisl Posted August 21, 2018 Author Posted August 21, 2018 (edited) 59 minutes ago, MWareman said: While it certainly would be nice is the Portal and ISY behaved the same - but it is correct what the Portal is doing and not correct what the ISY is doing. What you are trying to do should always have failed. The Portal is not 'processing the rest command' - nor is it 'not truncating'. Processing the full string is likely done by the Load Balancer (or other Amazon infrastructure) and is entirely correct. Truncating at the space is, per all standards, the incorrect thing (even though it's what allows your use case to succeed). It's a lack of processing code in the resource constrained ISY... I've said before.. adding the UOM to substitutions is the documented action. I've previously asked for a .RAW option on substitutions to direct the substitution to be made without the UOM added - but so far it's only applicable to the .ST value of nodes. If this concept were expanded to all substitutions that would solve this issue in a more sustainable way. If you have two ISYs - of course you could (as you document) send the values from one ISY to the other - relying on the truncation 'bug' to convert to an integer. Then you could send it back - or to Portal - now as an integer variable. I would caution that because you would be taking advantage of a likely unintentional 'feature' that it may stop working (or start functioning consistently) at some future version... I actually didn't say that any behavior was "correct" or "incorrect" I only said what the behaviors were. I'm also not sure why you are making a distinction between the "load balancer" and "the portal", the whole thing is "the portal", they are inseparable. Furthermore, there is nothing incorrect about saying the portal is not truncating the string variable into a numeric variable. This is a very important observation because of the fact that a REST to an ISY DOES. I did say that including units is more than just useless. It would be a crippling blow to the usefulness of the weather module if not for this "error" of the ISY REST handling. I also said that in its current form, you have to do a crazy workaround to use weather module data in anything more than the most basic and simple comparison (ie is temp greater than 80) Again, units are completely useless. You can add the units if you want to a notification email or text using isy's standard notifications format, or at the target device in a more generic sense. I have been using weather module data for a very long time via REST only and just always assumed it was only a numeric value, because 1) it made sense and 2) that is how it worked in that context. EDIT: I would also like to point out that UD is pushing their luck with the 128 character limit on path. The encryption string is huge and pushes very close to that limit. If memory serves, I was at 115 characters with the path using variable substitution. Edited August 21, 2018 by apostolakisl Quote
Michel Kohanim Posted August 21, 2018 Posted August 21, 2018 All, I have to take back my comment. The path can be as long as you want. There are no limits. With kind regards, Michel 1 Quote
apostolakisl Posted September 2, 2018 Author Posted September 2, 2018 On 8/21/2018 at 1:47 PM, Michel Kohanim said: All, I have to take back my comment. The path can be as long as you want. There are no limits. With kind regards, Michel Good to hear. This only leaves the issue of putting units into the variable for things like temp and humidity (and perhaps other things that I haven't tested). In my opinion, including the units by default is a bad idea. It is easy to append the units to a unit-less value when desired, but difficult to remove the units when not desired. Quote
Michel Kohanim Posted September 2, 2018 Posted September 2, 2018 Hi @apostolakisl, That's precisely the reason for node servers. The properties and commands have meaningful units associated with them and then you can use .raw for the value without units. With regards to the climate module, I am a little ambivalent mostly because the same underlying structure is used for programs and everything else. So, it'll be too intrusive to have separate paths for units vs. no units. This said, we are very much hoping to move the Weather Module to a node server. With kind regards, Michel Quote
apostolakisl Posted September 4, 2018 Author Posted September 4, 2018 On 9/2/2018 at 1:21 PM, Michel Kohanim said: Hi @apostolakisl, That's precisely the reason for node servers. The properties and commands have meaningful units associated with them and then you can use .raw for the value without units. With regards to the climate module, I am a little ambivalent mostly because the same underlying structure is used for programs and everything else. So, it'll be too intrusive to have separate paths for units vs. no units. This said, we are very much hoping to move the Weather Module to a node server. With kind regards, Michel I think it makes a lot of sense to have the weather module be a node rather than a module. I suppose you might say the same about the Elk module. Though if in the process of becoming a node it stops become a module, would that not require re-writing any programs associated with it? Quote
Michel Kohanim Posted September 5, 2018 Posted September 5, 2018 Hi @apostolakisl, ELK should also become a module. The main question is time/resources. For Weather, we are paying $$$$ monthly fees that we cannot recoup. And, therefore, that's our first priority. With kind regards, Michel Quote
apostolakisl Posted September 5, 2018 Author Posted September 5, 2018 1 minute ago, Michel Kohanim said: Hi @apostolakisl, ELK should also become a module. The main question is time/resources. For Weather, we are paying $$$$ monthly fees that we cannot recoup. And, therefore, that's our first priority. With kind regards, Michel Did you mean to say Elk also should become a NODE? Quote
Michel Kohanim Posted September 5, 2018 Posted September 5, 2018 @apostolakisl, Yes, that's correct: a node not a module. Apologies. With kind regards, Michel Quote
apostolakisl Posted September 5, 2018 Author Posted September 5, 2018 2 hours ago, Michel Kohanim said: Hi @apostolakisl, ELK should also become a module. The main question is time/resources. For Weather, we are paying $$$$ monthly fees that we cannot recoup. And, therefore, that's our first priority. With kind regards, Michel It just occurred to me, how does making it a node change any sort of access fee that would be involved for getting the weather data? I don't really know exactly where the weather data comes from or how it is billed. Is there a single one time charge for access that UD incurs with each subscriber? Quote
Michel Kohanim Posted September 6, 2018 Posted September 6, 2018 @apostolakisl, As a corporation, we had to purchase a business license. As an individual, you can get your own license. Just like the apps you can download on their website. With kind regards, Michel Quote
MWareman Posted September 6, 2018 Posted September 6, 2018 If rehashing the weather module, why not make it free/cheap for the module itself, then separate the data feed. User pays a monthly cost for a premium data source (like Aeries), or can use their own data feeds (a personal api key for WUnderground, a local meteobridge sourcing data from any number of station hardware, or even a local Weatherflow UDP data feed.... 2 Quote
Michel Kohanim Posted September 7, 2018 Posted September 7, 2018 @MWareman, Because each feed requires different API and code inside ISY. Polyglot will do the same without impact to ISY codebase. With kind regards, Michel 1 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.