Soft Keypadlinc-like Console from Raspberry Pi

[kck]

I just pushed V3.7 of the console to github as the current release.  The main motivation for this release was that the Pi4 doesn't handle frame buffers in the same way as all previous Pi versions with the result that the console may not orient in the same way as the desktop on at least some screens.  To address this and also add some general flexibility this release adds the ability for the console itself to rotate its screen relative to the underlying system's base orientation.  The release note is pasted below:

Soft screen rotation and streamlined install

This release primarily adds support for soft rotation of the display that is required for the Pi4, at least when used with the 7" Pi display. Pi 4 display drivers don't use the frame buffer in the same way as previous models so they may not honor rotations set at the system level. This results in the native desktop displaying in one orientation but the console displaying in another. The usage notes explain how to set the screen type to use soft rotation and the installation scripts now add support for this. (Note this has been tested with the Pi 7", the 28c PiTFT, and the 35r PiTFT screens at this point but should work generally.) This provides general purpose flexibiliy to arrange the console screen orientation as you like easily (and since it is set at console start changing the orientation is also very straightforward).

The console install scripts have been streamlined in this release to avoid the second reboot that used to be required. This should somewhat speed up installation although the bulk of the time is still in updating the base system to current.

The release also fixes a bug when an alert screen gets invoked while in Maintenance mode, a bug in MQTT status handling. It also reworks Home Assistant restart code to insure that states are correct for changes that happened while the console was offline. Finally it again adjusts the use of Adafruit install script use to handle their changes related to a 28c PiTFT.

[kck]

I just pushed a new release (3.8) that adds a photo frame capability at the request of a user.  (Something I had been thinking about adding for a long time but had been in procrastinate mode on).  Short blurb from github release below and full details added to the usage notes.

Photo Frame Screen

V3.8
This release adds a new screen type "Picture" to the console that allows the console to also act like a photo frame.  It supports a directory mode that sequentially displays jpg files found in that directory to the screen while that screen is active.  The directory is monitored for changes and those are reflected in the displayed photos within a few cycles.  It also supports a single picture mode that displays only a single jpg which is monitored for changes.  In this mode changes are reflected to the display within 3-4 seconds and so is suitable for use if the desire is to display changing screen captures or the like.  See the description in the usage notes for all the details on how it works.

The release also adds support for username/password for MQTT which was previously missing.  It adds an advanced network node command to the maintenance screens to clear retained knowledge of a dead node from the MQTT broker.  It also will handle MQTT server down situations more gracefully.  Finally, it adds a small developer oriented feature to enable personalized operations to be done on every console boot.  See the Developers Notes section of the usage notes for more information.

[kck]

Just pushed release 3.9 with a number of new capabilities  that had been requested by users.

File watch alerts and enhanced VARKEYs

This release has quite a few changes/enhancements:
FileWatch alerts and screens: It is now possible to set an alert that watches the modified timestamp on a file and triggers an alert when it changes. This is most useful in conjunction with an improved capability for alert screens. The message on an alert screen will now interpolate a store variable. The FileWatch alert will reflect the file contents into a store variable in one of two ways: either as a single store value that holds the entire file or as a set of variables within the FileWatch store that will contain the values set in the file. A simple example of use would be to generate an alert screen that displays the file contents when it changes. It is possible to clear such an alert automatically after a given time interval.

VARKEYs have been enhanced to be of more use in interacting with hubs. They now provide an option to invoke a program on the hub when pressed, and for HA hubs which allow parameters, pass a parameter. Their display options have also been enhanced. The net effect is that it is possible to construct a multibutton set of keys on a screen to do things like control a fan with speeds.

Related to the above, access to attributes of entities in HA can now be made via the store associated with the hub, e.g., HASS:light.ceiling:brightness would evaluate to the brightness setting of the light.ceiling entity in the HASS hub.

Weather fetching has been changed to operate autonomously for all defined locations. Previously, fetches were initiated when a screen needed them and there was not recent weather. Because it is now reasonable to use weather information in places other than weather screens, the console now simple updates weather whenever it expires, using the cache mechanism if multiple consoles are in use.

A minor addition is to allow semi transparent navigation keys if a picture screen is used as a regular screen rather than as an idle screen which improves appearance.

[mdfled]

Kevin,I am really interested in getting SoftConsole working, I think this is some fantastic work.  I have everything loaded and ran into an error seen in the console.log file snip below.  I loaded the mini-test version of the install. 

Here is the error.  

01-08-21 14:50:06 Sev: 3 SysParam: DimIdleListNames: ['MyClock']
01-08-21 14:50:06 Sev: 3 SysParam: DimIdleListTimes: ['20']
01-08-21 14:50:06 Sev: 3 SysParam: ErrLogReconnects: True
01-08-21 14:50:06 Sev: 3 Create ISY hub version 0 as My Lighting
01-08-21 14:50:06 Sev: 3 My Lighting: Create Structure for ISY hub at http://192.168.1.10 for user Mark
01-08-21 14:50:25 Sev: 3 My Lighting: Successful node read: 200
01-08-21 14:50:25 Sev: 5 Fatal console error - fix config file: 'folder'
01-08-21 14:50:25 Sev: 3 Traceback (most recent call last):
01-08-21 14:50:25 Sev: 3   File "console.py", line 455, in <module>
01-08-21 14:50:25 Sev: 3     hubs.hubs.Hubs[i] = pkg(i, v.get('address', ''), v.get('user', ''), v.get('password', ''), hubvers)
01-08-21 14:50:25 Sev: 3   File "/home/pi/consolestable/hubs/isy/isy.py", line 312, in __init__
01-08-21 14:50:26 Sev: 3     for folder in configdict['folder']:
01-08-21 14:50:26 Sev: 3 KeyError: 'folder'
01-08-21 14:50:26 Sev: 3 Console was up: 00mn 24sec
01-08-21 14:50:26 Sev: 3 Console Exiting - Ecode: 13
01-08-21 14:50:26 Sev: 3 Immediate Exit: 13
01-08-21 14:50:32 Sev: 3 End Log

 

I'm guessing the software didn't find the correct folder to read for the password for the ISY????  I checked the Auth.cfg file and it has the password to the ISY correctly listed along with the correct IP address and everything else looks correct.  Any idea of where the problem is?

One thing I'd like to change is my ISY is listed on the network as ISY99 and not my lighting, but I have left things for now as I'd like to get things working before I start making changes.

I am starting with the 7" display and an ADA fruit case but I have a couple of 3.5" screens in the wings waiting to get the bugs worked out.  

 

[kck]

No what you are seeing isn't related to your config files.  It is something about what the ISY is returning to the console.  Are all your devices in the ISY at top level?  I.e., do you have any folders defined to organize your devices in the ISY?  I think perhaps that the ISY doesn't return the folders list to me if there are none and I've never noticed that because everyone who has used the console has had at least one folder defined.  If that is what is happening I should probably handle that case correctly.

So - quick fix.  Try creating one folder in your ISY device list.  I doubt you even have to move a device into it.  See if that fixes things.  If it does, I'd just leave it there empty even if you don't want to organize things into folders and I will handle that "no folders" case in my next update just so you or others don't need to kludge it like this.  If, on the other hand, creating a empty folder doesn't fix it, try moving one device into the folder and see if that does.  If even that doesn't fix then we'll need to diagnose more.  Sorry for the problem.

 

Kevin

[mdfled]

Thanks Kevin,

So I added a folder in ISY called SoftConsole and tried to run SoftConsole and it failed.  I also tried to move a couple of devices into the folder as I was seeing an error that the values weren't integer values (progress?).  It still failed.  Attached is the console.log for your reference.

I really appreciate all the work that you have done and this looks very promising, the little bit of work I've had to do is nothing compared to what you've done, so no need to apologize.  I'm looking forward to the added functionality your project will add to my Insteon network!

 

Console.log

[kck]

OK - guess we'll have to look a little deeper at what your ISY is returning.  Could you edit your config file (probably config.txt) and right at the front of that file add the single line: 

ISYDump = True

Then run the console and you should find in your Console directory a file struct.dmp and send it to me.  There will also be a file xml.dmp that I don't think I will need but to save time in case I do attach that one as well.  Those should let me see exactly what the ISY is returning when queried as to it devices.  I hope that points directly at the issue.

I'll be out for a while starting a about half an hour so I may not get back to you until later tonight my time (Pacific Time) so depending upon where you are you may not see a response until tomorrow.

[mdfled]

Thanks Kevin,

Here is the struct.dmp file and the xml.dmp file.  Interesting I had to reboot the pi a couple of times to get it to come all the way up this time.  Looking at the struct.dmp file, I'm guessing, I have some devices that are structured oddly in the ISY???

struct.dmp xml.dmp

[kck]

Arg!  My original analysis was correct - I just didn't quite think it far enough in terms of Python and what the ISY was likely sending across.  This will sound really silly but create at least 2 folders and it should work.  Don't even think you need anything in them.

Longer version in case you are curious and do any programming.  The ISY sends across a json structure that Python internalizes that has all the node/folder information.  With no folders the ISY doesn't include any 'folder' field in the json.  That was your original situation that I had simply not realized would be the case.  The code expected that the internalization of the json would yield a dictionary entry for 'folder'.  Now part 2.  With multiple folders (the case the code expected) the value of the dict entry for folder is a list of folder items (which if you are a Python guy are dicts).  When there is only 1 folder the value of the dict entry for folder isn't a list of one folder item but instead is just that folder item.  The code wants to run through a list.  Boom!  That is why 2 folders should do the trick for now - it will cause a list to be created.

Definitely a bug in my code - I'll get a fix in place next release (which is probably next week since I have other things that I've worked on since the last update).

[mdfled]

Ok I added a second folder.  Now the raspberry pi goes through a cycle where I have just a cursor in the upper left, and there is no response from the mouse, keyboard, or the touchscreen.  The program seems to get caught by a watchdog timer as the desktop will reappear if I leave it sitting.  After a minute or two the console will try and start again and the screen goes back to no response with just the cursor in the upper left corner.  

I've added devices to both folders and I'm still getting the same cycle on the ras pi. I've also moved all my devices to the two folders and I still am getting the cycle.  

So console.log.1 is with no devices in the two folders, console.log.2 is with all the devices in two folders (SoftConsole1, and SoftConsole2).  I also included 3, 4, and 5 which should be logs of the pi cycling through the process.  Just out of curiosity (and to make it easy to break out of the loop) I removed the folders in ISY and the pi went back to the original error looking for a folder that didn't exist in ISY, and the pi did the short run and threw the original error. 

Console.log.1 Console.log.2 Console.log.3 Console.log.4 Console.log.5

[kck]

May be another case where I don't handle the null situation correctly and no user has had that situation to trigger the bug.  So - do you have integer variables but not any state variables in your ISY?  If that is the case, try defining a state variable (doesn't need any particular value or use - just needs to exist).  I think I am expecting that there will be at least a single state variable and a single integer (non-state) variable and am not handling the case when either type has no members.  If this fixes it, then I'll have to do a better job of noticing this case and allowing for it.

Again sorry - every new user always seems to exercise the ISY and console in slightly different ways and discovering latent bugs tends to be the result.

Kevin

[mdfled]

Thanks Kevin,

I'll take a look at this tomorrow.  I can't recall what variables I have defined on my ISY.   I'll recreate the folders in ISY and make sure I have both types of variables defined.

This reminds me of systems I used to work with; every customer wanted their own custom solution; which meant every time there was an upgrade; at least 1/2 the systems would crash because of all the different "customizations" which couldn't be tested.  

[mdfled]

Kevin,

We've made some progress, the cursor now changes about 30 seconds into the blank screen; since making sure I have both types of variables defined. I made sure there are at least 2 variables of each type; just in case the system wants more than one variable much like it's need for more than one folder.

Attached are the last couple of log files.  I think the current console log file is the last reboot which has the ISY back to no files; which I do so I can easily get back to the desktop to transfer the log files, so the console.log file may be irrelevant.

Console.log.3 Console.log Console.log.1 Console.log.2

[kck]

I want to make sure I understand what you are seeing.  What you should see is that after starting the screen turns blue and you see the boot log being displayed on the screen as the console comes up (this is the same entries that you see recorded in the Console.log files up through the message about arming alerts).  At the end of this initial log you would then see the screen change to the home screen you have defined.  If you aren't seeing the blue logging screen then there is something amiss regarding your screen settings for the console.  My best guess for that is that I notice that your console is using /dev/fb1 whereas my similar system (on a 7 inch screen) is using /dev/fb0.  Ever since Buster there has been something odd in the OS where some systems have an fb1 and some don't.  There is a kludge in the code to try to catch this but it may be that for your system fb0 is your touchscreen and fb1 is something else (perhaps an HDMI screen that was connected during system build).  In any case, if you aren't seeing the log, edit the file .Screentype in the home directory to delete the "B" at the end and that will turn off the check for the Buster kludge and should get you back on fb0 (which you should also see listed as the screen near the front of the log).

Separate from this, the console is seeing communications errors from your ISY.  These are not comm errors between the console and the ISY, they are comm errors between the ISY and the Insteon device that get reported in the event stream to the console.  The console attempts to prompt the ISY to recommunicate to such devices which is what the messages about the query attempts are.  It looks like the ISY gets through to the node but that it is having some difficulty doing so.  The message about ISY busy means that the ISY issued a busy status into the event stream that lasted nearly 22 seconds (looking at log.1).  That is kind of long which is why the console logs it.  Short busy periods are expected but longer ones not so often.  I mostly see those in my house for the 3am "query all" that the ISY does and for a evening and morning time when I have ISY programs that turn down the leds in bedroom switches - that takes a surprisingly long time to do.  The busy periods aren't a real problem other than that they create long lags for the console to actually turn a light on/off or the like.  But if you are seeing a lot of them it does suggest you may have noise on your power lines or that the device in question is just weak or distant from its nearest relay point.

Finally, it does look like you have an error in your config file of some sort.  The message about the key binding missing shouldn't be there.  This might be my fault if you are using the minimal example.  I did make some changes a while back to the install scripts that could have left a bug there - thought I tested it by perhaps it slipped by.  In any case, take a look at the config file and see if it looks right and if you can fix it post it to me and I'll take a look.

 

[mdfled]

Thanks again Kevin,

I got the console up finally.  I built another pi and it seemed to have trouble logging into the ISY, so I went back to the original SD card and edited the .Screentype file. I also put back in the original Config.txt file since you were seeing something corrupt (I'm in the habit of saving files with a new name if I make a change so that I can always revert to the original version).

Everything is up and working, now I need to configure things.  I'll do a little more testing to see if I have to have files in the two folders I created on the ISY.  I also need to figure out why the other build wouldn't connect.  I could see in the ISY error file that there was an authentication error.  The console log file wasn't even getting as far as the folder error I originally started out with, I suspect something is funny with the build.  I checked the Auth file and it looks like my credentials are correct.  I might send you my log files if I can't figure it out.  I'm assuming that I can have multiple softconsoles log into the ISY, but I guessed I needed to use a different device name so I called my second build SoftConsole2.  I'm assuming each one should have a unique name....

This has inspired me; One project I was working on was the raspberry pi garage door controller.  I lost interest since as it is configured, messages would be pushed to my smartphone.  I don't normally carry my smartphone around the house; and at night I don't bring it into the bedroom; with SoftConsole I should be able to glance at the screen and make sure the garage doors are down and all our exterior doors are closed.  I think the SoftConsole opens up more functionality, thanks for all your work on this.

[kck]

Great.  I'm happy to help if you run into any other issues.  I can't think of any reason that you should have to name the consoles differently that come from the ISY.   However, your router might have some objections to multiple instances of a name for DHCP and the like.  I name my consoles for the place they are located (as you can sort of see from the example configuration files).  Since the console will default to trying to look for its configuration file based on the node name, that lets me load all the configurations on to each console blindly and it will pick the one to use based on its name.  That is just a convenience since you can always just use config.txt as your config file.

If you do happen upon any cases where you figure out you had done something wrong but that the console log wasn't very helpful in figuring it out, I'm always interested in those case since I may be able to offer better error messages.  Once someone has the console up and running you don't tend to notice such things that could be improved so I'm really interested in new user experiences that can smooth out initial effort with the tool.

Re the garage door thing.  I do that at both of my houses.  At the one with the ISY, I use the Insteon garage door product which reports door status as well as allowing control of it.  I have alerts on most of my consoles in the house that start flashing at me if the door stays open for an extended period of time.  The alert will also let me close the door, although since you're not in visual contact with the door when you do that, it is a little dangerous.  At my seasonal place I use Zwave and Home Assistant as the hub and there I have a couple of cheap door sensors I got from Monoprice that report the status of the main and golf cart doors.  I use those to trigger alerts and also, since I have some zwave switches with LED status lights on them, also to alert via those. 

[kck]

I have just pushed a minor bug fix release 3.9.2 that addresses some of the issue above as well as some other things I happened upon.  Please note the note at the bottom of the blurb (think I probably blew out my getting old PLM on a power outage - arg.)

Mostly fixed minor bugs.  The console had expected incorrectly that there would always be folders defined in the 
ISY and that there would always be variables and scenes defined.  This is fixed in this release to allow these to
be empty.  Also, rather than shutting down and restarting if the console can't initialize a hub, it now replaces
the hub with a dummy hub that allows the console to start.  This caters to a situation I had where one of my hubs
became (semi) permanently unavailable and as a result consoles that have references to that hub wouldn't start to
allow access to other working hubs.  To reconnect to the off line hub the console does need to be restarted when
that hub again is up.

Some minor functional additions: this release supports scenes on the Home Assistant hub.  Note that HA scenes
can only be turned on (unlike ISY scenes).  A console button corresponding to a scene issues the corresponding
HA scene.turn_on command to the hub.  Previously, when displaying the log via the maintenance screen you needed
to scroll page by page to either the top or end of the log in order to exit the display.  Now a double tap on
the top half of the log screen goes directly to the 1st page of the log and a double tap on the bottom half
goes directly to the last page. From there a single tap exits.  Finally, the "Clear Matching Error" option
on the network command screen now behaves more usefully in that it issues the command to all consoles and the
match itself need not be completely identical for the match to work.  It need only be "close" thus handling
messages that hit all consoles that differ only in the address listed for some Python item.

Note: My own ISY system is currently down after a power outage and it will be a while before I can restart
it since it is in a different location.  Thus, the ISY changes in this release have had more limited testing
than I would normally do.  If you encounter any significant errors, you can revert to the previous version by
moving the "previousversion" directory in consolestable back up to be consolestable.  And of course report
the issue to me so I can deal with it.

[mdfled]

Thanks Kevin,

I've been working on getting my weather station interface, ntp server, and a couple of other items I want for the consoles, so I haven't had the chance to do much configuration of the SoftConsole yet.  I'll update my SoftConsole and install it on another machine probably later this week.  I'll let you know if I find any issues.

[mdfled]

Kevin,

I have a personal weather station running Weewx on a raspberry pi. The program is uploading my weather data to WeatherUnderground. I was wondering if you've played with Weewx at all?  Have you spent any time looking to see if you could receive the data from Weewx instead of using one of the commercial weather data streams.  There is a program within Weewx using a SDR receiver to pick up the transmitted data stream from a weather station so you may be able to pick up a neighbor's data stream if you don't have your own weatherstation.  It would address some of the issues as to how often you can get updates from the weather services. 

 

https://github.com/matthewwall/weewx-sdr

https://github.com/weewx/weewx

[kck]

I haven't played with it so don't really know anything too helpful.  However, the way the weather stuff works in the console is that the purpose of the weather provider stuff it really only to load stores with the current reported values so that they can be separately displayed by one of the screens.  A quick look at Weewx seemed to show that it can report information via MQTT.  If that is the case, then you should be able to define some entries like those in the example mqtt.cfg file that would pick off the values it was reporting and put them in the MQTT store.  Then you could display them as any other item in a store, in this case as something like MQTT:temperature.  That should allow you to show any Weewx information on the console.  I don't see any particular difficulty in doing this, but as always there might be some gotcha someplace that I'm not thinking of, so if you try it and run into issues let me know.

[Whitehambone]

@kck I am getting this error: 

 File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
root@raspberrypi:/boot# Exiting pisetup due to error in getinstallinfo

Any suggestions?  Is this a compatibility issue with my screen?

[kck]

No - nothing you did wrong.  It appears that the guys at Adafruit again altered their scripts without much notice and the shell script that I was calling has disappeared.  It looks like they now have a python script instead.  I will have to dig into that to see how to call it and make sure it leaves things as I need them.  Probably realistically can't get to this until Monday since I'm out much of Sunday.  I'll get on it and see how to fix it for their new stuff.

Sorry about this - not my fault but still annoying to see it happen and I'll try to fix ASAP.

Kevin

[kck]

@Whitehambone

I just pushed V3.9.3 to github which is a minor release that should fix the issue with Adafruit having changed their screen install scripts.  I've tested it with a 3.5 Resistive PiTFT and it worked fine.  It also should avoid any crash that occurred even if you were using some non-Adafruit PiTFT.  Let me know of any issues.

[kck]

V3.10 Dimming Support

I just pushed a new release of the console that has a significant new function in that it supports adjusting dim level for dimmers for both ISY and HA hubs.  Release blurb below and usage notes updated.

Significant release to add capability of using long presses and gestures on the touchscreen.  HA light domain nodes (dimmers) and ISY nodes of type 1 (dimmers) now have the added feature that holding a long press on their screen buttons will bring up a brightness slider.  Moving your finger along the axis of the slider will adjust brightness for the light.  Note that due to differences in how Insteon/ISY and HA/Zwave handle dimming the physical responsiveness of HA nodes may look quicker at the light itself.  This is due to rate limiting of the Insteon network traffic.  Tapping OK will return to the parent screen with the new brightness while tapping Cancel will return but return the brightness to what it was.  This feature also works for ISY Scenes via the console's scene proxy support so that the common idiom for implementing 3-way lights via a set of switches in a scene is supported as one would expect.  Note that on capacitive touch screens this works very well and very responsively.  Resistive touch screens generate a huge amount of "noise" in the form of non-existent touch/release events while holding down and other similar random things.  The code has a lot of "dejittering" for all of this with the result that the resistive screens may seem a bit more sluggish in their response to finger movement, but in my experience still quite acceptable.  The slider displays by default in the "long" direction of the screen but can be overridden, see the usage notes.  In addition to these changes, there is now an alternate way to bring up the maintenance screen via a single diagonal gesture from the upper left to the lower right of the screen.  (Note: if the screen is dim you do first need to awaken it with any touch.)  Note that this has required a significant revamp to the touchscreen handling code so if you do see any apparent weirdness please report it.  Also, given that the console can now see length of touch and movement, any suggestions for other uses of that capability are welcome.  Finally, in the event that you do see strange problems with touch, you can for now revert to the previous touch driver system by creating an empty file in /home/pi names .forceoldtouch.

[Greg]

Hey Kevin,

<edit> BTW Just North of you in Vancouver

I got the Softconsole installed and working.  Then I started trying to customize the cfg files, and what I was wanting to do was have a main screen that then branched off to the other screens.  I found the GOTO with the ScreenName =.  The first attempt at this seemed to work, so then I went back and tried creating all the screens I wanted. When I boot the console the error log says "GoTo Key target xxxxxx doesn't exist.  Not sure exactly what that means.  It says that all the files were merged.  Any hint or a code example would be great.  Below is just a snipet of what I have coded.

config.txt

include = Screen1.cfg UpStairs.cfg

---------------

Screen1.cfg

[Screen1]

type = Keypad

label = Main

[[UpStairs]]

type = GOTO

ScreenName = UpStairs

KeyColor = magenta

label = Upstairs Controls

----------------

UpStairs.cfg

[UpStairs]

type = Keypad

[[Dining Room Lights]]

KeyColor = orchid

label = Dining Room, Lights

type = ONOFF

[[Kitchen Lights]

 

 

Thanks,

Greg

Edited March 18, 2021 by Greg