Cheap Windows Jog/keyboard controller for CNCs

Overview

So we looked at the basic ways with hooks and RAWINPUT.

Windows has filter drivers, these basically sit on top of the existing drivers and can modify what the base driver does, simply a filter.

MSDN Filter Drivers

Software Needed

We will be looking at upper device level filters. As I mentioned previously MSDN is a great resource, what we're looking for now is the DDK or as its known now the WDK, Windows Driver Kit, this needs to be installed alongside Visual Studio 2013, grab it here.

WDK Downloads

The Windows 8 WDK works for Windows 7, which is what i'm using.

Debugger Notes

Grab all that, set it up. Note the remote debugging client, these are very useful for drive development, since we're at level where you're expected to know whats going on and be able to debug system exceptions, so a second machine or a VM is very useful. VMWare for instance has a Visual Studio plugin to help remote debugging inside a guest OS, great for low level work.

I also use a USB kernel debug cable for a second physical box, where i can't use or a VM isn't good enough.

Setting up a Kernel Debugger

You can use serial, firewire or USB for kernel debugging, but we likely won't need it for this.

Back to MSDN

Microsoft supplies code examples for almost every windows substem, and this no exception. We're looking at filter drivers, and there are ones for a PS/2 keyboard (kbdfilt) and a HID Mouse (firefly)

kbdfilt This is an upper device filter driver sample for PS/2 keyboard. This driver layers in between the KbdClass driver and i8042prt driver and hooks the callback routine that moves keyboard inputs from the port driver to class driver. In its current state, it only hooks into the keyboard packet report chain, the keyboard initialization function, and the keyboard ISR, but does not do any processing of the data that it sees. (The hooking of the initialization function and ISR is only available in the i8042prt stack.) With additions to this current filter-only code base, the filter could conceivably add, remove, or modify input as needed.

firefly Firefly is a KMDF-based filter driver for a HID device. Along with illustrating how to write a filter driver, this sample shows how to use remote I/O target interfaces to open a HID collection in kernel-mode and send IOCTL requests to set and get feature reports, as well as how an application can use WMI interfaces to send commands to a filter driver.

KMDF is Kernel Mode Driver Framework , WMI is Windows Management Instrumentation.

X64 Driver Signing and test sign mode

The WDK samples are setup to self sign, but X64 windows won't like it unless you boot in TESTSIGNING mode, which means adding a flag to the loader. http://www.ngohq.com/?page=dseo will test sign as well, and switch on testsign.

The reason i posted this entry before it was finished, is because even though i have my own cert, i forgot to sign it before installing it on this PC , and promptly killed my keyboard off, since it required a reboot to install it. Should be using a VM. You can run the virtual keyboard, which lets you get keypresses again if you do dev on a single box.

Setting up VMWare Player for driver debugging

After installing WDK into VS2013, setup a VMware Player with a wIn 7 x64, set a username/password and then under the driver/test menu in VS configure a new machine. Tell it the name of the VM ( make sure sharing is on in windows, firewall on too ) copy over and install the remote test sign tools, run those then run the auto config in VS

C:\Program Files (x86)\Windows Kits\8.1\Remote\ they're in this folder by default

Windows 8.x is really needed to use the network debugging features in VS2013.

VMWare by default doesn't pass in HID class devices, so to attach a keyboard or mouse directly to the VM , you'd need to add these to the VMX file

usb.generic.allowHID = "TRUE"
usb.generic.allowLastHID = "TRUE"

Then reboot or have the VM powered off. Now you'll be able to attach any HID devices directly to the VM , just don't attach your primary keyboard/mouse ;)

The kbdfiltr gets added to the UpperFilters registry key. Note that this adds the filter to ALL keyboard class devices, you can also add it to one specific device.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E96B-E325-11CE-BFC1-08002BE10318}\UpperFilters

kbfiltr
kbdclass

kbfilttr has to be before kbdclass. Just make sure its available and working before rebooting, otherwise the HID/keyboard system will fail to load. Though you can still use the old trick of loading the virtual keyboard.

Easiest way to add is use .reg file with

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\kbfiltr]
"Type"=dword:00000001
"Start"=dword:00000003
"ErrorControl"=dword:00000000
"Tag"=dword:00000006
"ImagePath"=hex(2):73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,44,00,\
  52,00,49,00,56,00,45,00,52,00,53,00,5c,00,6b,00,62,00,66,00,69,00,6c,00,74,\
  00,72,00,2e,00,73,00,79,00,73,00,00,00
"DisplayName"="WDK Keyboard Filter Example"
"Group"="Keyboard Port"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\kbfiltr\Parameters]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\kbfiltr\Parameters\Wdf]
"KmdfLibraryVersion"="1.11"
"WdfMajorVersion"=dword:00000001
"WdfMinorVersion"=dword:0000000b
"TimeOfLastSqmLog"=hex(b):23,9c,b2,fe,f2,7d,d0,01

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\kbfiltr\Enum]

then put the kbfiltr.sys in system32\drivers\

reboot the VM and if the keyboard is still working, run dbgview from sysinternals and press the numlock or caps lock a few times and you should see debug messages.

if not, check Test Mode appears in the bottom right of the desktop, and then the windows system log for issues with the driver.

Log of num lock press and disconnect, reconnect of my numpad. I added some more debug logs to my kbdfiltr source code

IOCTL_KEYBOARD_SET_INDICATORS
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_SET_INDICATORS
Enter: FilterEvtDeviceAdd 
Entered: KbFiltr_CreateRawPdo
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_INTERNAL_KEYBOARD_CONNECT
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_QUERY_ATTRIBUTES
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_SET_INDICATORS
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_SET_INDICATORS
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_SET_TYPEMATIC
Entered: KbFilter_EvtIoInternalDeviceControl
IOCTL_KEYBOARD_SET_TYPEMATIC

I made a desktop shortcut in the VM to the drivers folder, then i could just drag it across, you can also deploy from VS. Currently i just reboot it each change but we ought to be able to start and stop the driver to force a reload.

Ok so now if you're upto where I am, we're ready to do key substitution, lets do the Q to Z again.

open up kbflitr.c and look for KbFilter_ServiceCallback function

to make it easier since this is scan codes, add this DebugPrint to after this line

   devExt = FilterGetData ( hDevice );

DebugPrint ( ( "KbFilter_ServiceCallback 0x%x \n", InputDataStart->MakeCode  ) );

Scan Code List

add this after the DebugPrint, we won't get too fancy to start off with.

    switch ( InputDataStart->MakeCode ) {

        case 0x010:// 'Q'
            InputDataStart->MakeCode = 0x2C; // 'Z'
            break;
    }

Now open notepad and dbgview, and press the Q key. You should get something like this

I pressed Q 4 times, then W 4 times, then Z 4 times.

So there we go, keyboard filter, intercepting keys at the kernel level and replacing them.

All i have to do now is map the scan codes for my numpad and only install the filter driver on that device, and we're done!

Make codes

        Numpad make codes in hex

         789  - 47 48 49
         456  - 4b 4c 4d
         124  - 4f 50 51
         00.  - 52 52 53 (middle is marked as 000)
         /*-  - 35 37 4a
         +    - 4e
         BKS  - e
         ENT  - 1c

I'll map those to my CNC's keys and add modifiers, remember since we're dealing with scan codes here, ctrl shift alt etc don't mean anything yet. they're just scan codes same as (almost) any other key.

Not Invented Here / Reinventing the Wheel

Rather than do that, there is a project on codeproject talking about exactly this, and all the numerous pitfalls. It uses some of the API's we've already looked at.

Combining Raw Input and keyboard Hook to selectively block input from multiple keyboards

This more or less does what we want, but this method has a number of flaws. We can also look at using the non low level hooks, which requires a dll and hook injection (or access to the source code), look at WH_GETMESSAGE hooks for more information on that, since it combines both sets of information to make that determination.

System Wide Hooking for the WM_CHAR Message

So some solutions, not without their problems and one requiring hooking in into the apps, which may not work on all apps that use RAWINPUT, if we were just doing one app, and one machine this approach would likely work fine.

Roundup so far

So far we've looked at the global hook, which works really well but has the disadvantage of not being able to easily determine where the key came from and there are keys it can't catch, so it is basically a global search and replace. Then we looked at RAWINPUT which does tell us where it came from, but doesn't allow an easy substitution or removal of the messages. And now a project that married them together.

As we can see there are ways of doing this, if you're just looking for a quick working (mostly) solution, there are a number of programs available that use these principles, here are a couple

Keyboard Redirector

hidmacros /LuaMacros

AHK does it as well, if i recall, and uses the same methods, nearly all of these programs use a combination of the methods i've shown.

We've got enough to do this now

So if all you're looking for is a quick way to remap a second keyboard, stop here and grab those, enjoy your extra free time!

So is there a better way ? Yes. I'll cover this next update.

So in the previous example, it is a system wide low level keyboard hook that can change or remove key events from everything that generates output ( though some apps can bypass it) but what it doesn't tell us is where it came from. We need RAWINPUT for this. This will listen to nearly all the RAWINPUT's system wide.

Make a normal WIN32 project , make these changes.

i'll put these as complete examples on GitHub

Add crtdbg.h to stdafx.h for the debug macros.

Make InitInstance

//
//   FUNCTION: InitInstance(HINSTANCE, int)
//
//   PURPOSE: Saves instance handle and creates main window
//
//   COMMENTS:
//
//        In this function, we save the instance handle in a global variable and
//        create and display the main program window.
//
//        PostQuitMessage wont get the exit code out, since no pump, use debug RPT since no window yet.
//
BOOL InitInstance ( HINSTANCE hInstance, int nCmdShow )
{
    HWND hWnd;

    hInst = hInstance; // Store instance handle in our global variable

    hWnd = CreateWindow ( szWindowClass, szTitle, WS_OVERLAPPEDWINDOW,
                          CW_USEDEFAULT, 0, CW_USEDEFAULT, 0, NULL, NULL, hInstance, NULL );

    if ( !hWnd ) {
        return FALSE;
    }

    // raw device count
    UINT nDevices = 0;

    // used for result code
    INT nResult;

    // get number of raw devices, including fake ones for RDP etc, this allows  us
    // to allocate the right amount of memory, since we're only asking for how many are attached.
    nResult = GetRawInputDeviceList ( NULL, &nDevices, sizeof ( RAWINPUTDEVICELIST ) );

    // this would be bad. possibly permissions.
    if ( nResult < 0 ) {

        _RPT1 ( _CRT_WARN, "failed to get raw input device list  %d\n", GetLastError() );
        PostQuitMessage ( 4 );
        return FALSE;
    }

    _RPT1 ( _CRT_WARN,   "number of raw devices = %d\n", nDevices );

    // uhhh.
    if ( nDevices == 0 ) { PostQuitMessage ( 1 ); return FALSE; }

    // allocate enough memory for all the RAW devices reported

    pRawInputDeviceList = new RAWINPUTDEVICELIST[sizeof ( RAWINPUTDEVICELIST ) * nDevices];

    // failed ?
    if ( pRawInputDeviceList == NULL ) {

        _RPT1 ( _CRT_WARN, "failed to allocate memory for raw device list %d\n", GetLastError() );
        PostQuitMessage ( 2 );
        return FALSE;
    }

    // zero it
    memset ( pRawInputDeviceList, 0, sizeof ( RAWINPUTDEVICELIST ) * nDevices );

    // now get the actual list of raw devices
    GetRawInputDeviceList ( pRawInputDeviceList, &nDevices, sizeof ( RAWINPUTDEVICELIST ) );

    // failed ?
    if ( nDevices < 0 ) {

        // Clean Up
        delete[] pRawInputDeviceList;

        _RPT1 ( _CRT_WARN, "failed to get list of raw devices %d\n", GetLastError() );
        PostQuitMessage ( 3 );
        return FALSE;
    }


    for ( UINT i = 0; i < nDevices; i++ ) {

        // how much data do we need to allocate for the device name ?
        UINT nDeviceBufferLength = 0;

        nResult = GetRawInputDeviceInfo ( pRawInputDeviceList[i].hDevice,
                                          RIDI_DEVICENAME,
                                          NULL,
                                          &nDeviceBufferLength );

        if ( nResult < 0 ) {

            _RPT1 ( _CRT_WARN, "failed to get raw input device name length %d\n", GetLastError() );

            continue;
        }

        // allocate memory for device name (wide char)
        WCHAR* wcDeviceName = new WCHAR[nDeviceBufferLength + 1];

        // failed?
        if ( wcDeviceName == NULL ) {

            delete[] pRawInputDeviceList;
            _RPT1 ( _CRT_WARN, "failed to allocate memory for raw device names %d\n", GetLastError() );
            PostQuitMessage ( 5 );
            return FALSE;

        }

        // get the RAW device name
        nResult = GetRawInputDeviceInfo ( pRawInputDeviceList[i].hDevice,
                                          RIDI_DEVICENAME,
                                          wcDeviceName,
                                          &nDeviceBufferLength );

        // failed?
        if ( nResult < 0 ) {

            _RPT1 ( _CRT_WARN, "failed to get actual raw device name %d\n", GetLastError() );


            // free the memory for the name
            delete[] wcDeviceName;


            // skip this one
            continue;
        }

        // setup buffer to receive device info
        RID_DEVICE_INFO ridDeviceInfo;
        ridDeviceInfo.cbSize = sizeof ( RID_DEVICE_INFO );

        // this tells the api how big the structure is
        nDeviceBufferLength = ridDeviceInfo.cbSize;

        // fetch the device info for this raw device
        nResult = GetRawInputDeviceInfo ( pRawInputDeviceList[i].hDevice,
                                          RIDI_DEVICEINFO,
                                          &ridDeviceInfo,
                                          &nDeviceBufferLength );

        // failed?
        if ( nResult < 0 ) {

            _RPT1 ( _CRT_WARN, "didn't get raw input device info %d\n", GetLastError() );

            // skip to next device
            continue;

        } else

            // is it a keyboard?
            if ( ridDeviceInfo.dwType == RIM_TYPEKEYBOARD ) {

                _RPT2 ( _CRT_WARN, "device handle 0x%x (%d)\ndevice name ", pRawInputDeviceList[i].hDevice, i + 1 );

                // wchar type
                OutputDebugStringW (  wcDeviceName );

                _RPT1 ( _CRT_WARN, "dwKeyboardMode %d\n" , ridDeviceInfo.keyboard.dwKeyboardMode );
                _RPT1 ( _CRT_WARN, "dwNumberOfFunctionKeys %d\n" , ridDeviceInfo.keyboard.dwNumberOfFunctionKeys );
                _RPT1 ( _CRT_WARN, "dwNumberOfIndicators %d\n" , ridDeviceInfo.keyboard.dwNumberOfIndicators );
                _RPT1 ( _CRT_WARN, "dwNumberOfKeysTotal %d\n" , ridDeviceInfo.keyboard.dwNumberOfKeysTotal );
                _RPT1 ( _CRT_WARN, "dwType %d\n" , ridDeviceInfo.keyboard.dwType );
                _RPT1 ( _CRT_WARN, "dwSubType %d\n" , ridDeviceInfo.keyboard.dwSubType );
            }

        // don't need this anymore
        delete[] wcDeviceName;

        wcDeviceName = NULL ;

    }

    // setup the global RAWINPUT device

    ZeroMemory ( &rawInputDevice, sizeof ( rawInputDevice ) );

    /// ignore legacy messages, and
    // RIDEV_INPUTSINK = If set, this enables the caller to receive the input even when
    // the caller is not in the foreground. Note that hwndTarget must be specified

    rawInputDevice.dwFlags = RIDEV_NOLEGACY | RIDEV_INPUTSINK;

    //  https://msdn.microsoft.com/en-us/library/ff543477.aspx
    rawInputDevice.usUsagePage = HID_USAGE_PAGE_GENERIC;

    rawInputDevice.usUsage = HID_USAGE_GENERIC_KEYBOARD;

    // needs to know where to send the message, when using RIDEV_INPUTSINK 
    rawInputDevice.hwndTarget = hWnd;

    // we want WM_INPUT's
    nResult = RegisterRawInputDevices ( &rawInputDevice, 1, sizeof ( rawInputDevice ) );

    // failed?
    if ( nResult == FALSE ) {
        _RPT1 ( _CRT_WARN, "register raw input devices failed %d\n", GetLastError() );
    }

    ShowWindow ( hWnd, nCmdShow );
    UpdateWindow ( hWnd );

    return TRUE;
}

when run it'll scan through all the RAWINPUT devices and list out, via the debugger, the id/guid for them , as well as any details on the device itself, we're only looking at keyboards and you'll also see any virtual keyboards listed. It then registers for RAWINPUT messages with RegisterRawInputDevices. This requires a change to WndProc

Add this to the message switch

        case WM_INPUT:

            // wParam is either RIM_INPUT (this app foreground) or RIM_INPUTSINK (this app background)
            // lParam is the RAWINPUT handle

            // determine size of buffer
            if ( GetRawInputData ( ( HRAWINPUT ) lParam,
                                   RID_INPUT, NULL, &dwSize, sizeof ( RAWINPUTHEADER ) ) == -1 ) {
                break;
            }

            LPBYTE lpb ;

            // allocate it
            lpb = new BYTE[dwSize];

            if ( lpb == NULL ) {
                break;
            }

            ZeroMemory ( lpb, dwSize );

            // get actual data
            if ( GetRawInputData ( ( HRAWINPUT ) lParam,
                                   RID_INPUT, lpb, &dwSize, sizeof ( RAWINPUTHEADER ) ) != dwSize ) {
                delete[] lpb;
                break;
            }

            // process it
            {

                PRAWINPUT pRaw = ( PRAWINPUT ) lpb;
                WCHAR wcTextBuffer[512];
                UINT keyChar = MapVirtualKey ( pRaw->data.keyboard.VKey, MAPVK_VK_TO_CHAR );

                wsprintf ( wcTextBuffer,
                           TEXT (
                               "Type=%d\nDevice=0x%x\nMakeCode=0x%x\nFlags=0x%x\nReserved=0x%x\nExtraInformation=0x%x\nMessage=0x%x\nVKey=0x%x\nEvent=0x%x\nkeyChar=0x%x\n\n"
                           ),

                           /// device header
                           pRaw->header.dwType,

                           // device handle, pass this to GetRawInputDeviceInfo
                           pRaw->header.hDevice,

                           pRaw->data.keyboard.MakeCode,
                           pRaw->data.keyboard.Flags,
                           pRaw->data.keyboard.Reserved,
                           pRaw->data.keyboard.ExtraInformation,
                           pRaw->data.keyboard.Message,
                           pRaw->data.keyboard.VKey,
                           keyChar );

                OutputDebugStringW ( wcTextBuffer );
            }

            // not needed
            delete[] lpb;

            break;

This grabs all the RAWINPUT events we registered before and dumps them to the debugger window.

So this shows the underlying RAWINPUT and gives us much more information, we can see which device the input came from, but unfortunately we can't change it easily here. pRaw->data.hDevice contains the handle to the device the message came from, so we can filter by device. .Useful for looking at individual RAW device output

MSDN is an amazing resource for Windows programming

Using Raw Input

Make a win32 console project in Visual Studio or so.

add windows.h

in main() add

printf ( "setting global hook\n" );

// Insert a low level global keyboard hook
HHOOK hook_keyboard = SetWindowsHookEx ( WH_KEYBOARD_LL, ll_keyboardproc, 0, 0 );

// now just the windows message pump
MSG msg;
while ( !GetMessage ( &msg, NULL, NULL, NULL ) ) {
 TranslateMessage ( &msg );
 DispatchMessage ( &msg );
}

// unhook keyboard hook
UnhookWindowsHookEx ( hook_keyboard );

add this function, this gets called on every key press, up and down events (some keys need special handling)

/**
* ll_keyboardproc
*/
LRESULT CALLBACK ll_keyboardproc ( int nCode, WPARAM wParam, LPARAM lParam )
{
// if keyup event , gets set to keyup flag
int keyup = 0;

// are we replacing/removing the key
bool replace_key = false;

// process it, if its an HC_ACTION otherwise pass on to OS
if ( nCode == HC_ACTION ) {
// key up/down etc
switch ( wParam ) {

case WM_KEYUP:
case WM_SYSKEYUP:
keyup = KEYEVENTF_KEYUP;
// falls thru
case WM_KEYDOWN:
case WM_SYSKEYDOWN:

PKBDLLHOOKSTRUCT p = ( PKBDLLHOOKSTRUCT ) lParam;
switch ( p->vkCode ) {

//  process individual keys
case 'Q':
// send our replacement key (as an up or down message)
keybd_event ( 'Z', 0, keyup, 0 );
//remove the original key press
replace_key = true;
break;

case 'Z':
keybd_event ( 'Q', 0, keyup, 0 );
replace_key = true;
break;
default:
break;
}
break;
}
}

// pass on to OS
if( replace_key == false ) {
return CallNextHookEx ( NULL, nCode, wParam, lParam ) ;
}

// remove the original key press.
return 1;

}

formatting gets removed in the code box.

this will replace Q with Z and Z with Q, or q/z. You could also do something else, call a function, run an exe, output a digital line etc. Obviously this is also a simple key logger, you can also send multiple key presses, so now its a macro playback etc. there is some additional logic for sys keys etc, but keeping it simple.

so pretty simple, install a global keyboard hook and process the messages. this would work if you had one keyboard and wanted to remap the keys, but it is less useful.

next is using RAWINPUT to determine where the key came from