Let's talk about mouse control. Since before Myo even launched, developers have been able to easily enable mouse control in Myo Scripts. That's been very powerful and convenient, letting people build some really interesting connectors that just wouldn't have been possible otherwise.

Since then though, one of the most common features we've been asked for is a way to do mouse control in full applications. Well, that day has arrived. We've open sourced the code we are using in Myo Connect to control your cursor both as an example and for easy integration into other projects. Let's dive in.

If you're using the Windows or OSX SDK and just want to use the default mouse control, you can use this source files (MouseMover.hpp, MouseMover.cpp, LinearParameter.hpp and Utilities.hpp) directly. Get started by importing them into your project, and #include MouseMover.hpp. For example, here's a stripped down version of the Hello-Myo sample:

// Copyright (C) 2013-2014 Thalmic Labs Inc.
// Distributed under the Myo SDK license agreement. See LICENSE.txt for details.
#define _USE_MATH_DEFINES
#include <cmath>
#include <iostream>
#include <iomanip>
#include <stdexcept>
#include <string>
#include <algorithm>
#include <myo/myo.hpp>

// MouseMover is where all the magic happens
#include "MouseMover.hpp"

//This is for mouse movement on Windows
#pragma comment(lib, "user32") // or link to the library normally, this gets it done in one file for the sample
#include <Windows.h>

// Classes that inherit from myo::DeviceListener can be used to receive events from Myo devices. DeviceListener
// provides several virtual functions for handling different kinds of events. If you do not override an event, the
// default behavior is to do nothing.
class DataCollector : public myo::DeviceListener {  
public:

    DataCollector()
        : mouse()
    {
    }

    // onOrientationData() is called whenever the Myo device provides its current orientation, which is represented
    // as a unit quaternion.
    void onOrientationData(myo::Myo* myo, uint64_t timestamp, const myo::Quaternion<float>& quat)
    {
        mouse.onOrientation(quat);
    }
    void onGyroscopeData(myo::Myo* myo, uint64_t timestamp, const myo::Vector3< float > &gyro)
    {
        mouse.onGyroscope(gyro);
        std::cout << '\r';

        // Print out the change in mouse position
        float dx = mouse.dx();
        float dy = mouse.dy();
        std::cout << "dx: " << std::setw(5) << dx << "    dy: " << std::setw(5) << dy;
        std::cout << std::flush;
        moveMouse(dx, dy);
    }

    // Tell our MouseMover which way the Myo is being worn
    void onArmSync(myo::Myo* myo, uint64_t timestamp, myo::Arm arm, myo::XDirection xDirection, float rotation,
        myo::WarmupState warmupState)
    {
        mouse.setXTowardsWrist(xDirection == myo::xDirectionTowardWrist);
    }

    // Windows mouse movement code
    void moveMouse(float dx, float dy) {
        INPUT input = { 0 };
        input.type = INPUT_MOUSE;

        input.mi.dx = (LONG)dx;
        input.mi.dy = (LONG)dy;

        input.mi.dwFlags = MOUSEEVENTF_MOVE;
        SendInput(1, &input, sizeof(INPUT));
    }

    MouseMover mouse;
};
int main(int argc, char** argv)  
{
    try {
        // Set up myo. See standard hello-myo.cpp
        myo::Hub hub("com.example.hello-myo");
        std::cout << "Attempting to find a Myo..." << std::endl;
        myo::Myo* myo = hub.waitForMyo(10000);
        if (!myo) {
            throw std::runtime_error("Unable to find a Myo!");
        }
        std::cout << "Connected to a Myo armband!" << std::endl << std::endl;
        DataCollector collector;
        hub.addListener(&collector);
        while (1) {
            // In each iteration of our main loop, we run the Myo event loop for a set number of milliseconds.
            // We aren't doing anything outside the loop, so this value doesn't super matter.
            hub.run(1000 / 20);
        }
        // If a standard exception occurred, we print out its message and exit.
    }
    catch (const std::exception& e) {
        std::cerr << "Error: " << e.what() << std::endl;
        std::cerr << "Press enter to continue.";
        std::cin.ignore();
        return 1;
    }
}

To use it, all we need to do is pass the orientation data and gyroscope data into our MouseContoller as it comes in. The actual calculation happens with the gyro data, so you'll want to read dx and dy right after that. The value you get back is the amount of pixels to move. In this sample, we print it out to the screen and send it to the Windows SendInput function to actually move the mouse.

The code is designed to act as a black box, but the source is there if you want to poke into it and either port it to another platform (like Unity), or tweak other parameters (of which there are a fair number). Basically, this is what happens though:

  1. Convert gyro data to the world frame of reference. The gyroscope gives you a vector representing the angular velocity of the user's arm in degrees per second. The problem main problem is that it's from the point of view of the gyroscope, which is in the pod with the logo. We need to rotate that into "world orientation" so it lines up with the screen. That involves a bit of quaternion math, but it's probably not something you'll need to modify.
  2. Extract the x (left/right) and y (up/down) parameters of the gyro vector. We don't care about z (roll). We pass those values into updateMouseDeltas() function, which is what we're really interested in.
  3. Determine the gain. This is the relationship between arm movement and pixels. Basically, what's the factor that we multiply x and y by to covert from degrees per second into pixels. The big factors are sensitivity and acceleration, which have handy functions you can tweak any time. There are a bunch of others (like pixel density) you can tweak in the code itself, but they often just act as a constant scaling factor, or are bounds on parameters like acceleration.
  4. Calculate and store the number of pixels to move. Multiply x and y by the gain (and the frame duration). Since pixels are integers, we accumulate any fractions on the side, truncate the original, and if we have enough leftover fractions to make a whole number, we add it back in.

Again, since the number of pixels to move is just stored in MouseMover and not actually acted upon, you need to retrieve before the next time you call onGyroscope(), and then use that to update the mouse position yourself.

It's worth mentioning that cursor acceleration is a key part of this algorithm. It really works best on a large screen from a few meters away, but you can tweak the sensitivity and acceleration values to make more sense for your application (and even provide a slider for the user to set themselves, like Myo Connect). However, it's definitely not a 1:1 arm-to-screen-position mapping. If you want that (which may be appropriate for a lot of cases), it's just a matter of projecting the orientation of the user's arm onto the screen.

Making a 1:1 mapping isn't hard, but we'll save that for another post. For now, enjoy this powerful mouse control algorithm!

See you next time!

Newsletter

Enter your email address and get all latest content delivered to your inbox every now and then.