Where's My Keyboard?

After you experience this problem for the first time it seems very obvious: when you put on a head-mounted display you are going to need to start whatever VR environment you want to play with using your keyboard or mouse. But you can't see your keyboard and mouse anymore because the display is blocking your vision. What do you do? Most people grope around awkwardly for a bit before finding the right device.

After starting one of the virtual environments you are usually set. Either the experience does not require user input (movie-like), or some extra input like head-tracking, a Kinect, or voice control is activated. However, the current selection of VR interaction modes is very limiting because it eschews the interaction channel that we are most accustomed to when working with computers: text. Though this can be a wonderful benefit to a virtual reality, letting us forget that we're just having our buttons pushed by a number-crunching machine, there are lots of interesting applications the textless input situation neglects.

Motivating Example: Creating virtual worlds

Project Philosophy: This video is what started my mind churning on this project. As a programmer whose also an artist, the possibilities for creative construction within virtual reality excite my imagination like nothing else I see in technology today. Usually when I'm making things there's a hard wall between the work that I do with my hands, and the work that I do with a computer. I'm forced to use the "physical knowledge" stored in my muscles in an entirely different context than the abstract knowledge I use to interact with computers, even though all of the work has the same goal. Virtual reality offers a chance to break down that divide. It could let me design and build digital and physical objects with my hands and mind at the same time.

Once virtual reality masters both interaction that is conventional for the physical world (human body movement) and interaction that is conventional for the existing computer world thanks to projects like Keychange, a threshold will be crossed at the edge of a revolution in how computers are used as tools by humans. I'll be able to put on goggles, pick up Keychange, and sculpt something like a robot with my bare hands (making use of interaction that is intuitive for that purpose). At any moment I can switch to modifying its behavior using interaction intuitive for the computer (text). Then I can see how well it works immediately and modify the design as needed. When I'm all done I can have the computer send the designs to places all over the world, which will fabricate the parts, maybe assemble them, and ship the result back to me.

I think it's clear why this would be revolutionary. The design-build loop shrinks close to its limit (assuming teleportation or matter compilers aren't available to revolutionize shipping) and the pace of technology will increase a thousandfold.

Project Logs

Collapse

Prototype: Shorty
Owen Trueblood • 01/05/2015 at 18:32 • 0 comments

Shorty's purpose is to be a platform for developing the firmware that will translate button presses into keypresses on a PC. I did not start working on it with the intention giving it a life beyond dev work on Keychange, but it has turned out to be so useful to me in general that I might develop it further into a stand-alone project. The killer use is supplying tactile input while using tablets.
My primary mobile computer is an MS Surface Pro. It does not have a built-in keyboard, and the floppy "cover" keyboards are terrible, in my opinion. You have to use a kickstand to hold the screen up when typing on them, which only has one angle (too vertical for comfortable use) and is really uncomfortable to use on your lap. So I use a super cheap netbook for typing work like programming. But when I design PCBs I love using Eagle on my Surface, because manipulating eagle with a pen is so much more pleasant than using a mouse. But if you have ever used Eagle, you know that there are lots of (weird / unintuitive but) useful keyboard shortcuts and using them is essential to efficient PCB work. Before Shorty I had to waste half the screen on the software keyboard and put up with the annoyance of it being slower to manipulate than a hardware keyboard.
With Shorty I can stop using the software keyboard. I threw together a super simple firmware that just has the 8 most common Eagle shortcuts mapped to the buttons. I stick Shorty to the back of my Surface where my left hand goes. Then I have the pen in my right hand. And altogether it makes a super powered PCB design tool. It is a really specific use-case, but such a helpful one for me that it will make all the extra work of fleshing out Shorty worthwhile.
Prototype: Giraffe Grip
Owen Trueblood • 01/05/2015 at 18:09 • 0 comments

Working out the lesson I learned from my previous grip prototypes, that comfortable typing will only be possible with rigid support, I spent a little bit of time mocking up a grip design that braces against the forearm. It is bulky, but it feels much more comfortable than the palm-braced designs I have been making so far. I might make a second version that is lighter, adjustable, and foldable to see if it could be a practical approach for the final design.
PCB 1 Assembled
Owen Trueblood • 01/05/2015 at 17:38 • 0 comments

The PCB that I designed and ordered from OSH Park arrived. The design is in the Github repository for the project. I have assembled it, applied a minor fix (forgot to route traces for the programming header), and tested it with some basic firmware.
The design uses an RN-42 for Bluetooth communications. It is what lets Keychange emulate a Bluetooth keyboard and mouse. On the other side of the board is an NRF24L01+ 2.4 GHz transceiver module. It will be used for communication between two Keychange devices so that one can be used per hand without wires connecting them. In the middle is the Atmega32 that runs the show. To the side are connections for the buttons (there are extra connections for prototyping), for the joystick, and for the battery module (a LiPower boost converter).
Early Grip Experiments
Owen Trueblood • 01/04/2015 at 23:48 • 0 comments

The ergonomics of Keychange have been getting a lot of my attention. It is very important to me that this device is comfortable to use, because if it does its job successfully then it will be in my hands all day. If I get the shape and mechanics of it wrong then I will have a device that is unpleasant to use and possibly even one that could lead to injury (repetitive strain injuries are no joke for people who need to interact with computers all day). So I have been spending about as much time working on giving it a pleasant shape as working on the electronics and programming components that will make it actually work. Key to the effort are physical prototypes that I can hold in my hands.
Playdough Grips
For a while I was sketching out designs on paper and making mock-ups in 3D modelling software. This is usually how I approach the initial design of physical objects. But the sketches were difficult to produce because the shapes I wanted were organic and 3D. It was hard to convey the full shape with just lines in pencil or pen. 3D modelling helped with that problem, but it took much more time and could not let me keep up with the ideas I wanted to play around with.
So I quickly decided to try doing the modelling by hand. Clay would probably be the best material for the task, but I did not want to make the trip to the art store or spend a lot to order it online. I remembered from 1st grade that it is really easy and cheap to make playdough. Basically all you need is flour, salt, and oil. I already had all those things, so I made a big batch and spent a while making different comfortable shapes in my hand. My playdough was not nearly as good at holding specific shapes as clay would have been, but it did a great job of giving me intuition for what good shapes might be. I tried baking the prototypes shapes I made in the oven to harden them, but bizarrely the insides of them turned liquid and oozed out, leaving only a thin shell behind. They were still useful, but not very resilient.
I liked one of them in particular because it was small and especially comfy to hold. I took some Cherry MX switches and pushed them into little square holes that I cut into the playdough shell. It looked pretty weird, but it gave me some more ideas about how it might feel to press buttons while holding a shape like that. Unfortunately I learned that it is really fatiguing to press buttons on a shape where all of the support is coming from the base of the palm. The fatigue comes from the effort required to hold the thing still while pulling a finger down. My take-away was that the final design should have really good bracing against the wrist or even part of the arm. Just bracing against the hand will lead to wiggle room that makes typing feel bad and leads to more energy being expended per key press.
Laser Cut Grips
After messing around with playdough I decided that it would be useful to do prototyping in a material strong enough to allow me to mount buttons and electronics and have a working mock-up. So I turned to laser-cut 5 mm plywood. Working in 2D was much faster than working in 3D, but was not as flexible and the designs that resulted were not really comfortable. I explored two designs in plywood: a palm-braced minimal grip and a gun-style grip.
I look at both of them as failures because they are not that comfortable to hold. And the gun-grip looks too much like a gun. It probably would be bad to have that kind of association tied to the final design. My take-away lesson is that hand-made fully 3D prototypes are the best when exploring ergonomics.
The process of finding an optimal shape for Keychange has just begun, and many more physical prototypes are ahead.
Prototyper: First Data
Owen Trueblood • 10/30/2014 at 20:27 • 1 comment
Yesterday I finished the Prototyper hardware and wrote software to collect information on timing between keypresses. The firmware I wrote accepts commands over serial that specify a first key and a second key. Then it sends back the number of milliseconds between when the first key is pressed and when the second is. A Processing sketch running on a PC prompts the human being tested to press specified randomly chosen key sequences. It records the data collected from the hardware to a CSV file.
Data
After finishing the firmware and PC software and working out a couple of bugs I did a test run on myself. I recorded about 1000 key sequences (~2000 key-presses). Then I wrote a Python program to generate the matrix graph below, where average timing for every key sequence is normalized and color coded. The left-hand vertical axis is the "from" key, and the horizontal axis is the "to" key. The numbering starts with 0 at my right hand pinky and goes to 9 at my left hand pinky.
Observations
The data is noisy and I suspect that there is not yet enough of it to draw strong conclusions. But some preliminary observations can be made:
- The yellow spattering in the upper right seems to indicate that it generally takes me longer to go from my right hand to my left hand.
- My left pointer finger is pretty fast overall, which is strange because I am right handed.
- My right pointer finger is slow if the next key is on the left hand, but generally fast otherwise.
- Pinkies are the overall fastest. I think this has to do with the weird way I learned to type on QWERTY keyboards.
Improvements
Having data is exciting, but I need to make sure it really tells me what I need to know before relying on it. There are a couple of problems I need to resolve before I use Prototyper to collect much more data:
- While collecting those thousand samples I noticed that it took me a lot longer to get ready for certain key sequences. Because I was only recording the time between the first press and the second, that setup time was lost. For the future I will collect that time on the PC side and record it.
- The data gathering PC application indicates key sequences with colors. Green is the first button to press and red is the second. Often I would simply make the mistake of messing up the order of presses. This reflects a poorly designed GUI, not problematic sequences, so I'm going to replace colors with an arrow that pulses in the right direction.
- Just taking the mean of times for a key sequence when analyzing the data is probably a bad idea because a single outlier can mess up the whole set for a key sequence. I don't know much about statistics, but friends of mine do, so I'll get their help in future analysis.
- I should be recording when completely incorrect keys are pressed, because that is also indicative of a problematic sequence.
The Prototyper & Arpeggiators
Owen Trueblood • 10/27/2014 at 22:59 • 1 comment

It's a Testing Platform
The goal of this little project is to have a basic general purpose platform for trying out key-to-symbol mapping schemes.
Last night I desoldered some keyswitches from a junked keyboard, took some measurements of my hand on cardboard, threw together a laser-cut design in SolidWorks, cut the design out of 5 mm plywood, hot-glued the pieces together, popped in the keys, soldered together a messy per-key pull-down circuit on a perfboard, plugged the mess into an Arduino, wrote some firmware to send 2 bytes with key state every time the state changes (debounced), and wrote a basic Processing sketch that uses java.awt.Robot to send keypresses to my OS. With those basics in place I can start playing with different ways to translate button presses into characters or control keys.
First Test Subject: Arpeggio Input
In my last project log I laid out problems that I see with the input model used by chording keyboards. I am collecting evidence that shows chording keyboards to be inherently slower than their traditional counterparts, due to greater mental and physical effort when executing them. I believe that a solution is to put arpeggios in the place of chords.
Not a Chorder, an Arpeggiator
An arpeggio is a "broken chord" where the notes of a chord are played one-after-another instead of together. In keyboard terms, to produce a character like '~' on an arpeggiating keyboard (term I just made up) you might press key 1, then key 2, and lastly key 4, where on a chording keyboard you would press keys 1, 2, and 4 at the same time. Even though common sense would dictate the former to be slower than the latter, because the time taken to press all the keys at once would probably take the same time as pressing just one key in the sequence, a closer inspection makes the situation less clear. It can be quite difficult to coordinate multiple fingers at the same time. Anybody who has learned to play chords on a guitar or even a piano knows this. With practice it can be done, but it is still slower than carrying out an action with one finger. In the hand and arm, the machinery of the fingers is physically linked. Try moving each of your fingers independently. You will notice that it is impossible to keep all other fingers still while moving one. I think this hints that more strength, focus, and muscle memory is required when executing chords.
Pressing buttons one at a time can be done extremely quickly, as we know from using traditional keyboards. Interestingly, I see anecdotal evidence that speed scales when decreasing the number of keys and the distance between them. This video of an extremely fast calculator user, and my own experiences programming on calculators in the past, inspired in me the idea that this might be the case. My hypothesis is that a keyboard which assigns one key to each finger can require multiple independent key-presses per character and still be as fast as a traditional keyboard. The fingertips do not need to translate through space, so the basic time unit is that required for a finger to depress a key. It seems quite possible that in less time than it would take to remember and execute a chord a user could remember and execute an arpeggio.
I do not yet have any data to back up my hypothesis, but that is why I built the Prototyper.
The Mental Difference
I had a eureka moment when I thought about exploring arpeggios for Keychange. Since the inspiration came in part from my experience programming for many hours per day on my trusty TI-83+ calculator, I pulled my old calculator out of storage and programmed on it a small prototype of the idea. I made use of 4 keys and decided on using the basic lower-case alphabet. Organized into a tree where each node leads to 4 others (one branch per button) all of the leaves (letters) could be reached in 3 key-presses. I set about learning to type "hello". Surprisingly, it only took a few minutes. Learning to type any letter took half an hour. After that I felt like the idea needed...
Read more »
Why Chording Keyboards Never Win
Owen Trueblood • 10/27/2014 at 21:42 • 1 comment
I'm still thinking hard about the basic problems inherent to the success of an input device like Keychange. I believe that the combinations of solutions that past alternative keyboard projects used is mainly what relegated them to the historical dust bins. In this update I'm going to lay out what I think those problems are and how I intend to solve or avoid them.
Existing Chording Keyboards Are:
1. Unfamiliar - Looking at most chording keyboards, a non-techy person might have difficulty in even recognizing that the device is a keyboard. Each appears as an organically shaped high technology object covered in buttons. The gut reaction is fear and ridicule. Fear like fear of math. A person looks at some formulation and are told it is very useful, but the symbols used to describe it give an air of great complexity. People fear the unknown, so they avoid the device in favor of an easier alternative. Ridicule because such devices will be seen as tools for nerds who must care too much about the pointless, boring problem of digital text input.
2. Memory-Heavy - The basic premise of a chording keyboard is found in its chords. For every symbol there is a button combination that summons it. Unfortunately, the inherent trade-off is that a much higher mental effort is needed to get the benefits of the device. Most people will get discouraged when they find out they need to memorize dozens of chords before they can take advantage of a device. The traditional keyboard also requires practice, but a person can start using it immediately because every button is labeled with what it will do. For the traditional keyboard, efficiency is proportional to time. For a chording keyboard efficiency is zero for some time, and then it jumps up and slowly increases thereafter. The difference is a crucial friction point that prevents the wider adoption of chording keyboards.
3. Slow - For the bright, nerdy people who have traditionally used chording keyboards this is likely the most severe problem. Chording keyboards are inherently slower than their many-keyed counterparts. In reviewing the experiences of many chording typists I have found that it is not simply a matter of practice. Like in playing traditional musical instruments, motions involving individual fingers are much faster than motions involving the coordination of multiple fingers. Part of this problem is mental - it takes longer to order around several fingers compared to one. And part of the problem is physical - the machinery of our fingers is not independent per digit. Some fingers move together, and that makes many chords inherently more difficult to execute.
4. Bespoke - With unfamiliarity, this is a problem not inherent to the concept of chording keyboards. Alternative keyboards of all types have never seen wide popularity. So people who want them are forced to reinvent the wheel and come up with a custom solution. For those people who actually go and do it this is not actually a bad thing, I think, because they are more likely to be the type who would want a custom solution anyway. But when unlike people go looking for alternative keyboards they mostly just find those custom projects and face a wall. They do not have the skills to reproduce one of the projects, do not want to take the time, do not see a project that fulfills their needs, or cannot find enough details about any attractive project to reproduce it.
How Keychange is Different, and Why it will Win
Problem: Unfamiliar
Familiarity is a matter of context. Traditionally alternative keyboards have been fighting for the same space as the incumbent typewriter-style keyboard. Juxtaposing the two makes the alternative look like a loser. Most people are not going to bet on an underdog in a situation where they are mostly happy with the status quo.
Keychange sidesteps this issue of juxtaposition through virtual reality. In the context of virtual reality it is actually conceivable that the incumbent's advantageous position has been weakened and an underdog could...
Read more »
New Direction: VR
Owen Trueblood • 10/17/2014 at 22:13 • 1 comment

In the four months since initially putting up my "Chord Keyboard" project on Hackaday.io I've not done any physical work on it. Partially that's because of other projects and schoolwork, but mostly it's because I just haven't prioritized it. Chord keyboards have been built over and over since the dawn of computing (and even before then) but they've never caught on because the standard keyboards work well enough for most people. Every time I'd consider working on the project, I'd decide that my time was better spent working on things that lots of other people would find utility in.
But recently I was thinking about the project and had a breakthrough. I was trying to imagine an argument with some other person where my aim is to convince them to use my chording keyboard. The benefits that I see in the device were just not good enough to convince the other person to go through the effort of using my device over the existing standard solutions.
So my mind wandered to other contexts where the existing solutions might be worse off. And that's when I started considering chording keyboards in relation to virtual reality. It's an area of development that I've been following closely after trying out the Oculus Rift last year. In the time since that (really cool!) experience I have been exposed to lots of awesome virtual reality demos and the relayed experiences of friends who have interned at Oculus or played with modern virtual reality systems.
Taking all of that together, I am convinced that virtual reality is going to be a big deal sometime in the near future. And there's a little niche for me to carve out within it by developing a really good keyboard for virtual reality. If I succeed, then a much larger group of people will benefit from my work compared to if I had simply added another chording keyboard to the pile of attempts. And I can personally obtain all the benefits that I wanted when I started down this road.
So I'm going to put a lot of effort into designing this keyboard for virtual reality. To start it off, I'm acquiring ~$500 in funding from my school (the program is called TechX, for the curious). In exchange, I'll develop a polished prototype by the end of January and present it to fellow students and the representatives of many technology companies. Last year I did the same for a different project and ended up in a booth next to Oculus Rift. Hopefully that happens again this year...
Reflecting the changed direction and renewed vigor, I've removed the old project logs. They are archived here: owen-t.me/keychange/paleotype.html