#MyoCraft: Painting! And Keyboard Mapper Scripts

I like Paint.NET. It's a free graphics program for Windows. Nothing fancy, but it has layers, filters, and a few tools. It's also way easier than GIMP. I thought some gesture-enabled painting would be fun so I whipped up a quick set of key mappings for it with Myo Connect's build in Keyboard Mapper. For this weeks #MyoCraft, we're going to talk about the resulting script. Yes, that's right, if you make some mappings you like, you can export them into the Application Manager so they are automatically active whenever you use that application later. How does it do that? By dynamically writing a Myo Script for you! That's pretty cool on it's own, but there are some neat techniques being used in the scripts themselves as well. Let's dive in!

Using the keyboard mapper was pretty straightforward. I set fist to click and hold the left mouse button, fingers spread to undo, wave left to select my brush, wave right to select the eraser, and double tap to toggle mouse and unlock.

I exported it, making sure it was only active on the Paint.Net application. This was the result:

  

scriptId = 'com.thalmic.keyboardmapper.Paint.Net'  
    minMyoConnectVersion = '0.9.0'  
    scriptTitle = 'Paint.Net'

    function activeAppName()  
        return 'Paint.Net'
    end

    ----------------------------
    -- Helpers
    function conditionallySwapWave(pose)  
        if myo.getArm() == 'left' then
            if pose == 'waveIn' then
                pose = 'waveOut'
            elseif pose == 'waveOut' then
                pose = 'waveIn'
            end
        end
        return pose
    end

    unlockType = ''  
    function unlock(type)  
        unlockType = type
        myo.unlock(unlockType)
    end

    function getUnlockType()  
        return unlockType
    end

    keyPressSuspendedUnlockTimer = false  
    function keyPress(key, edge, ...)

        if edge == 'down' and getUnlockType() == 'timed' then
            unlock('hold')
            keyPressSuspendedUnlockTimer = true
        end

        myo.notifyUserAction()
        myo.keyboard(key, edge, ...)

        if edge == 'up' and keyPressSuspendedUnlockTimer then
            unlock('timed')
            keyPressSuspendedUnlockTimer = false
        end
    end

    mouseClickSuspendedUnlockTimer = false  
    function mouseClick(button, edge, ...)  
        if edge == 'down' and getUnlockType() == 'timed' then
            unlock('hold')
            mouseClickSuspendedUnlockTimer = true
        end

        myo.notifyUserAction()
        myo.mouse(button, edge, ...)

        if edge == 'up' and mouseClickSuspendedUnlockTimer then
            unlock('timed')
            mouseClickSuspendedUnlockTimer = false
        end
    end

    -- Toggles unlock and keeps mouse control synchronized with unlock state
    enabledMouseOnUnlock = false  
    function unlockAndControlMouse()  
        unlock('hold')
        enabledMouseOnUnlock = true
        myo.controlMouse(true)
    end

    function onLock()  
        -- If mouse was enabled on unlock then disable
        -- it regardless of how the lock was issued
        if enabledMouseOnUnlock then
            enabledMouseOnUnlock = false
            myo.controlMouse(false)
        end
    end

    function onDeactivate()  
        -- Disable mouse control so that we always resume
        -- this script with mouse control disabled
        if enabledMouseOnUnlock then
            enabledMouseOnUnlock = false
            myo.controlMouse(false)
        end
    end

    ----------------------------
    -- Map poses to functions
    LOCKED_BINDINGS = {  
        doubleTap_on = function() unlockAndControlMouse() end
    }

    UNLOCKED_BINDINGS = {  
        waveOut_on = function() keyPress('e', 'press') end,
        waveIn_on = function() keyPress('b', 'press') end,
        fist_on = function() mouseClick('left', 'down') end,
        fist_off = function() mouseClick('left', 'up') end,
        fingersSpread_on = function() keyPress('z', 'press', "control") end,
        doubleTap_on = function() myo.lock() end
    }

    function currentBindings()  
        if myo.isUnlocked() then
            return UNLOCKED_BINDINGS
        else
            return LOCKED_BINDINGS
        end
    end

    function onPoseEdge(pose, edge)  
        pose = conditionallySwapWave(pose)
        fn = currentBindings()[pose .. '_' .. edge]
        if fn then
            fn()
        end
    end

    ---------------------------
    -- Script activation handling
    function onForegroundWindowChange(app, title)  
        return platform == 'Windows' and 
               string.match(app, 'PaintDotNet.exe')
    end

    ---------------------------
    -- Set how the Myo Armband handles locking
    myo.setLockingPolicy('none')

    function onActiveChange(isActive)  
        if not isActive then
            onDeactivate()
        end
    end  

This was my first attempt at painting with my Myo armband. You can tell it's art because it says so right there in the picture.

This should look fairly familiar if you are used to our function binding approach. Unlike our previous examples, however, it's actually USING the binding system to have two separate sets of bindings: LOCKED_BINDINGS and UNLOCKED_BINDINGS. This, naturally, makes it easy to have a different set of bindings for when the Myo armband is locked or unlocked. We do this to let you set any gesture to unlock your Myo armband. The correct binding to use is determined in the currentBindings function based on whether the Myo armband is unlocked or not:

 
function currentBindings()  
        if myo.isUnlocked() then
            return UNLOCKED_BINDINGS
        else
            return LOCKED_BINDINGS
        end
    end

function onPoseEdge(pose, edge)  
        pose = conditionallySwapWave(pose)
        fn = currentBindings()[pose .. '_' .. edge]
        if fn then
            fn()
        end
    end

So if it's locked, it will return LOCKED_BINDINGS. If not, you get UNLOCKED_BINDINGS. Then we pull the appropriate function out of that list based on what pose and edge comes in. If it's a valid function (ie, an unlock pose if it's locked, or a pose bound to lock or a key press if it's not), we call it.

Here is my second attempt. Note the subtle use of like, 8 colours. The roof is orange because I couldn't figure out how to make a good brown. Let's just say it's not the Myo holding me back at this point.

The other useful thing I'd like to point out is the inline function definitions. For example, here's LOCKED_BINDINGS:

  
 
LOCKED_BINDINGS = {  
        doubleTap_on = function() unlockAndControlMouse() end
    }  

The function for doubleTap_on is being defined right there in the array itself. We can do this because functions are first class values in Lua. Remember that the normal function definition syntax of something like

  
function getUnlockType()  
        return unlockType
    end

is actually just some syntactic sugar for getUnlockType = function () return unlockType end, which is exactly what were are doing in the array. Doing it this way saves some space and makes the script generation less complicated. You may not necessarily want to write your OWN scripts like that, but it certainly does simplify typical pose-key press actions.

So that's how the sausage is made. The Keyboard Mapper isn't doing anything that you couldn't already do with a Myo Script, it just makes it dead simple to map poses to key presses. It doesn't touch the IMU at all, but with a bit of knowledge there is no reason you couldn't start with a basic Keyboard Mapper script and add some motion controls yourself.

This is why they pay me to write. I had a lot of fun painting, but when I asked one of our artists to give it a try she used her Myo armband to whip up some actual art.

That's it for now! Don't forget to submit your projects to MyoCraft@thalmic.com.

Otherwise, see you next week!