Programmable Keyboard for Less and Windows UI Automation

I made a a sliding drawer for my keyboard, and while at first I thought that I would not further derail myself from actually practicing, you know, making music, a brief experience with LMMS[a] sent me right away doing things other than actually doing music. See, the keyboard is not just a bit in front of my "typing" keyboard and my mouse - it's also offset a bit to the left. This makes it quite a reach to get at the mouse when I needed to do something in LMMS, and of course that's a job for the hobby engineer to solve.

First I thought I'd get a "macro keyboard", like this eight-key programmable keyboard[b], but then I realized that it wasn't programmable enough. I wanted much more than eight functions, which meant that I'd need some kind of bank-switching where some key or keys would change the function of the other keys. If I had bank-switching, I needed some kind of display to show what the keys actually did.

I've also been bombarded with ads for Maschine+[c] (probably because of my search and/or browsing history​[1]), and the multi-function pads on the Maschine+ made me realize that I didn't need a physical keyboard as much as I needed a programmable touch display. But where to get a touch display?

Why, everyone has at one of them nowadays. We call them cellphones.

For the basic architecture of the system I wanted to go with something simple: I'd start a server process in Windows and use the browser on the cellphone to display the UI. The server would receive events from the cellphone and use the SendInput API to simulate keystrokes. Said and done, I went ahead and implemented it.

It worked just as I had expected. I could send any keystroke (and event mouse movement and clicks) to LMMS. I played around with the F5, F6, F7 and F8 keys to show and hide the various editor windows in LMMS, and everything was just perfect. Then I realized that LMMS doesn't have any keyboard shortcuts for, well, a lot of important things. I could send any keyboard command LMMS supported to it, but it just didn't support the keyboard commands I wanted it to support.

Back to the drawing board.

In my next attempt I tried to walk the UI element hierarchy, trying to find the buttons that I wanted to press - but I only got as far as the top-level windows before the UI element hierarchy abruptly ended. Turns out LMMS doesn't use Windows controls. Then Inspect[d] gave me a hint: UI automation[e]. The UI elements I wanted had UI automation enabled and were visible in that element hierarchy.

Long story short: UI automation worked. It was possible to find elements and invoke actions on them. I now had a nice programmable keyboard that I could put on the desk above the musical keyboard and control LMMS through.

2021-10-30 11:49

c++, diy, java, LMMS, music, tech

Vattugatan 15

The biggest problem was really to figure out how to go about it. COM isn't exactly easy to use. So here's a very short tutorial, maybe it'll help someone:

// Initialize COM. We'll use the apartment (single threaded) model.
CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);

// Get a reference to the UI automation service
IUIAutomation* automation = NULL;
HRESULT hr = CoCreateInstance(
    __uuidof(CUIAutomation),
    NULL, CLSCTX_INPROC_SERVER,
    __uuidof(IUIAutomation),
    (void**)&automation);
    
if (FAILED(hr)) {
    // error
    return;
}

// The UI element tree is rooted in the desktop
IUIAutomationElement* desktop = NULL;
HRESULT hr = automation->GetRootElement(&desktop);
if (FAILED(hr)) {
    return;
}

// Do something (see below)

desktop->Release();

Once you have an automation instance and a root element, you can start by walking the tree. This is how you list direct child elements to an element:

void iterateChildren(
         IUIAutomation* automation, 
         IUIAutomationElement* parent) {
    IUIAutomationTreeWalker* controlWalker = NULL;
    IUIAutomationElement* element = NULL;

    automation->get_ControlViewWalker(&controlWalker);
    if (pControlWalker != NULL) {
        goto cleanup;
    }

    controlWalker->GetFirstChildElement(parent, &element);
    if (element == NULL) {
        goto cleanup;
    }
            
    while (element)
    {
        // Do something with "element".
        
        IUIAutomationElement* next;
        controlWalker->GetNextSiblingElement(element, &next);

        element->Release();
        
        element = next;
    }

cleanup:

    if (controlWalker != NULL) {
        controlWalker->Release();
    }

    if (element != NULL) {
        element->Release();
    }
}

To walk the whole tree you have to use iterateChildren recursively by calling it in the // Do something with "element".-line. Then you can get the type and name of the element by calling the get_CurrentControlType and get_CurrentName.

IUIAutomationElement* element = ...;

BSTR name;
element->get_CurrentName(&name);

CONTROLTYPEID controlType;
element->get_CurrentControlType(&controlType);

These are basically the only two reliable properties you can use to identify a UI element. (There are other properties, but these two are the ones you'll have to work with.) Once you have the element you need to know which Pattern it implements. There are a number of patterns listed on MSDN[f]. Note that there is no requirement for a certain CONTROLTYPEID to implement the corresponding pattern (an element with type UIA_WindowControlTypeId does not necessarily support IUIAutomationWindowPattern) - you must test whether the pattern is supported. This is how you can test for the Invoke pattern and use it:

IUIAutomationElement* element = ...

VARIANT available;
element->GetCurrentPropertyValue(
    UIA_IsInvokePatternAvailablePropertyId, 
    &available);
if (!available.boolVal) {
    // Handle case when pattern isn't available
}

// Get a pattern COM interface to the element
IUIAutomationInvokePattern* invocation = NULL;
HRESULT hr = el->GetCurrentPatternAs(
    UIA_InvokePatternId, 
    IID_IUIAutomationInvokePattern, 
    (void**)&invocation);
if (SUCCEEDED(hr)) {
    // Do the Invoke call.
    hr = invocation->Invoke();
    if (SUCCEEDED(hr)) {
        // Invocation success
    }
    invocation->Release();
} else {
    // Handle failure
}

Footnotes