Capture what Echo hears into a variable!

[jratliff]

UPDATE 3/8/16: Below is the original post of how I creatively found a way to send speech from Echo to Tasker. I now figured out how to make a skill as well that is much cleaner if you're interested in that too: https://www.youtube.com/watch?v=cSLdjHmf9K4

 

As many of you have probably wanted from the Echo, I wanted to be able to say any command and have that directly interpreted into my specified actions, without having to invoke a skill or use any limited Amazon commands and be able to chain commands...

 

Well, I've done it!

 

https://youtu.be/FOi9UkBKIgE

 

Short story:

I monitor the Alexa history page for change on a PC, copy the speech Alexa posts there when changed, use EventGhost to save it in a variable and send it on to Tasker for full interpretation and call ISY tasks or any other task!

 

 

Long story how to:

 

I noticed when viewing your history on the alexa.amazon.com website (go to settings then history) it almost instantly pops up whatever it heard. I got to thinking there had to be some way to copy that text as it comes up and send it off to be acted on by your desired third party. First I tried just highlighting the text but that is blocked on the site. So after some searching and trying a few programs I found capture2text. This is an on screen OCR reader. You press the hot key to start it, highlight an area of the screen, then click to capture that area. It converts it to text and can do several things with it via its options. I opted to copy it to the clipboard and prepend echo=:= to the captured text (this is in preparation to send it to Tasker on my always on Android box, =:= is a command operator for Tasker).

 

Next I installed EventGhost. I have always heard about it but never really used it. Played around with it and installed the Tasker AutoRemote plugin. This plugin allows EventGhost and my Android devices to communicate. I set up a macro in EventGhost as follows:

Send Message Macro

1) Event trigger - clipboard change (capture2text copies to clipboard, the macro is triggered)

2) Python script - (the following script opens the clipboard, assigns it to a variable, closes clipboard)

 

import win32clipboard

 

# get clipboard data

win32clipboard.OpenClipboard()

eg.event.payload = win32clipboard.GetClipboardData()

win32clipboard.CloseClipboard()

3) AutoRemote message - send {eg.event.payload} as a message to my android device (eg.event.payload is the variable we saved the clipboard contents to)

 

 

Next we want capture2text to highlight the proper area and capture the text on its own. I created another macro in EventGhost as follows:

Capture Text Macro:

1) Event trigger - (more on this next) EventGhost command prompt command ("C:\Program Files\EventGhost\EventGhost.exe -event Echo_Activated"). This pops up the event "Echo_Activated" in EventGhost and starts the macro.

 

2) Move the mouse pointer to: x:21, y:265 (this moves to top left corner of text to copy)

3) Emulate Keystrokes: (Shift+Q) (this is the hotkey to start capture2text capture box)

4) Wait: 0.5 sec (this makes sure capture to text got the command and is ready)

5) Move the mouse pointer to: x: 650, y: 294 (bottom right corner of text to copy, box should be formed around it)

6) Wait: 1.5 sec (allows capture to text to register the area)

7) Left mouse button (clicking signals to capture2text to capture the area we just highlighted, this will also fire the previous macro we set up since text was just copied to the clipboard)

Wait: 1.0 sec

9) Start Program: Pixel Check Echo.exe (this I figured out last and is explained below)

 

 

Now we need a way to automate telling EventGhost when new text has popped up. First I was going to take the echo apart and wire an Insteon I/O linc to the leds so that it could look for text every time Alexa was called on, but I didn't have the right torx screwdriver so I stopped after taking the rubber bottom off the Echo (I did order a set of torx though my next project is to have hue lights come on blue around the house when Alexa is listening so I'm still going to try wiring in an I/O linc).

 

After a lot of searching and trial and error I found this pixel check script someone had posted. It was written in AutoHotKey. Basically you specify an area of the screen to watch, it calculates a checksum for that area and compares it however often you like. I set it to 300ms. When a change is detected there are a number of things you can write in the script to do but I just opted to run the command line operation noted earlier that signals EventGhost to run the Capture Text Macro.

 

Pixel Check Program:

So here is the script that you need to put into AutoHotKey which can then be compiled into an .exe file and run independently. Every time the program detects a change it exits and needs restarted which is where the line 9 comes in above causing it to start watching the area of the screen that you'll have your Alexa History window running. Link to where I got the script: https://www.autoitscript.com/autoit3/docs/functions/PixelChecksum.htm

 

 

AutoHotKey script:

#cs ----------------------------------------------------------------------------

 AutoIt Version: 3.3.14.2

 Author:        

 

 Script Function: Monitor an area of the screen for changes then notify EventGhost of a change.

#ce ----------------------------------------------------------------------------

 

; Script Start - Add your code below here

#include <MsgBoxConstants.au3>

 

 

; Wait until something changes in the region 21,265 to 200,285

 

; Get initial checksum

Local $iCheckSum = PixelChecksum(21, 265, 200, 285)

 

; Wait for the region to change, the region is checked every 100ms to reduce CPU load. (I set mine to 300, testing what works fine)

While $iCheckSum = PixelChecksum(21, 265, 200, 285)

    Sleep(300)

WEnd

 

;Uncomment the next line if you want a message box to pop up, I used this for testing initially but don't want a pop up normally.

;MsgBox($MB_SYSTEMMODAL, "", "Something in the region has changed!")

 

;Run eventGhost event

Run ("C:\Program Files\EventGhost\EventGhost.exe -event Echo_Activated", "")

 

;I was going to use this to send an AutoRemote message to EventGhost originally, I just left it here commented out in case anyone ;found it useful to see how to call a url. Could send a rest command etc.

 

;Call url if changed

;$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")

;$oHTTP.Open("GET", "", False)

;

;$oHTTP.Send() 

 

 

I'm not going to go into using Tasker as that's a whole other story. But essentially I have a Task listening for an AutoRemote message of "Echo" (remember I prepended "Echo=:=" to the text Alexa heard). The part after =:= is saved in the Tasker task as a variable, by default this is %arcomm. I then use the action "AutoVoice test command" and input %arcomm as the command. This sends whatever Alexa heard to Tasker's AutoVoice. Then you can set up all kinds of great profiles to interpret the text received.

The nice thing about AutoVoice is you can setup substitutions for commonly misheard words, write multiple ways to say the same thing (start music|play music|turn on music) or (tv|t.v.|t. v.), chain commands, break out individual words, it's almost endless... Anyhow, Tasker also has a Mobilinc plugin or can call REST commands on the ISY which is the endpoint for most of my Tasks.

 

So that's it! Seems like a lot but it really isn't too bad. Hard part was finding all the tools and scripts. I wasn't familiar with most of these programs, just did a lot of searching and reading to get what I needed.

 

I've changed this setup around several times as I've begun using it. I'm sure this could be done more cleanly or more efficiently so I welcome thoughts, comments, and suggestions!

 

Jason

 

 

Required Software

EventGhost - http://www.eventghost.net/

Tasker AutoRemote Plugin - http://joaoapps.com/autoremote/eventghost/

Capture2Text - http://capture2text.sourceforge.net/

AutoHotKey - https://www.autohotkey.com/

 

And I used this site to capture mouse x,y position on my screen (I resized the window and moved it's capture box over the area I wanted to click, alt+tab to my alexa.amazon.com window and position mouse, alt+tab back to this site and click without moving mouse. It gives you the x,y coordinates where you clicked

 http://www.w3schools.com/jsref/tryit.asp?filename=tryjsref_event_mouse_screenxy_clientxy

[MWareman]

Clever!

[blueman2]

Wow.  A Rube Goldberg process to get there, but I really think this is great!  This opens up a lot of opportunities for doing very neat things with Alexa without developing a skill.  Well done!!

[jratliff]

Thanks! It has been working pretty well. I am able to use "play" "pause" "stop" and "turn up tv" commands among other duplicates built in as long as "play" isn't followed by anything it finds to actually play. "Turn up tv" however does cause the echo to turn itself up too.

 

I added a few more macros to EventGhost as well. One to start everything back up when you reboot. Another to restart the computer weekly to help keep it from hanging on occasion. And another to turn the screen off at night.

 

I'm working on setting up a profile in Tasker to send text messages too so I can just ask Alexa to send the text!

[jratliff]

Also interesting, I'm just playing with an old phone connected to the echo's bluetooth, if I use the "say" command in Tasker it plays through the echo. So all my verbal confirmations can play back through the echo too!

[Javi]

Nice work!

 

I have not purchased an ECHO as I think being able to access the entire voice command is the only way I could use it without it feeling like a toy.  I wish Amazon would just let us have access to the command without going though everything you had to do.

 

Anyway, I wanted to suggest a few things for your tasker setup.  I have setup a project (in the tasker section of this forum) that will eliminate the need for (start music|play music|turn on music) or (tv|t.v.|t. v.) and use a single regex with a list of your devices, actions, and locations. This will also give you the ability to say things naturally and chained together for example "Turn on the kitchen lights" or "Turn the kitchen lights on" and chain commands with chains linked to actions and devices for example "turn on the lights and the tv" and "turn off the kitchen, dining room, and living room lights". It has a default room option as well .  I have posted the tasker xml to github but have also created an app, it would not be hard to add AutoRemote as a trigger for the project.

 

Also the same program can be written in python to use in EventGhost...It would probably take me a couple of hours to write if I leave out the UI of the Android app. So you could setup the command links with the android UI then copy the backup  from Android to EventGhost, This way you could eliminate one step which would help with latency and eliminate another possible point of failure.

 

I'm curious, can you get the same info from the amazon app on your android device? If so, Joao may be able to add this to AutoVoice (or another AutoApp) as AutoVoice is essentially reading the screen of the Google Now Voice Search Box, and I'm Sure it would not be hard to have it read the screen of an Amazon app. 

[jratliff]

I came across your project a few weeks ago I hadn't dug into it much yet since I had a lot of Auto Voice profiles made long before I had the echo so I was just using them but I was definitely interested in checking it out more when I had time! Looked like a nice smooth setup for commands.

 

And yes the Alexa app updates the same as the web, I had no idea how to pull the info from there so I went with the pc to start. I was hoping someone knew how to either monitor the web page in the background without the need for the ocr stuff and screen on. But it would be awesome if it could be pulled directly from the app running in the background.