Litar: LiDAR Air Guitar

Here's a quick video of the Litar in action. You can see the Litar is working with battery power over Bluetooth LE MIDI to FL Studio (through a loopback driver since FL Studio doesn't support Bluetooth MIDI yet).

There's still a fair bit of work in the coming weeks to getting a sensible chord and scale mapping for the strings and buttons, as well as velocity and aftertouch effects to actually play some decent music, but the main hardware components and string detection algorithms are working as intended.

In the quest to reduce the latency of string measurements, it was discovered that there is significant sensor parallax due to each sensor measuring independently of one and other (clock drift over time, different start times). Notice the out of phase measurements below from two different sensors:

This means that it may take up to 20ms (at 100Hz) to be able to operate on the latest sensor measurements from all sensors when detecting strings.

To counter this, the MappyDot firmware required the addition of a synchronisation pulse to keep measurements in line with each other. The pulse is sent along the addressing lines from one device to the next (as soon as the pulse is received it sends it to the next device in the interrupt routine to reduce propagation delay).

The master device controls the firing of each measurement when in synchronisation mode. So if it is operating continuously, or in single shot mode, each subsequent device in the chain will also fire at the same time. It has the added benefit of only requiring a single "take measurement" command to be sent to the master, when in single shot mode.

So with this in place we get the following synchronised measurements without the parallax (measurement pulses shown from two random sensors):

Yay, 10ms string measurements!

The Litar boards are populated and ready for testing:

Looking good!

We finally have some boards for testing! The previous Litar prototypes were just a jumble of wires and protoboards.

These boards contain a LiPo charger circuit, BLE module, hall effect sensors and USB connectivity. They connect up to the string of MappyDot Pluses with a small cable.

It is hoped to release these as a kit sometime :D

Here's a bit of a taster of the strings working on the Litar. The LEDs light up when the corresponding string is being touched.

There's 4 strings so it's the 4 middle sensors that light up (they are manually turned on with the LED on and off commands of the MappyDot).

So it turns out, contrary to the information on the ST Microelectronics site, the VL53L1x sensor actually does run up to 100Hz at shorter ranges (<1m) pretty reliably after some testing (it was mentioned this was possible in some of their older press releases).

Which means a halving of the sensor latency to 10ms. This is absolutely awesome!

As part of the new button case design, the Litar will also be getting an upgrade to the MappyDot Plus we have been testing, which uses the brand new VL53L1x sensor:

The new sensor offers better ambient light immunity and an adjustable field of view. This will hopefully allow us to change the size of the strings programmatically and offer better string strumming recognition in noisy environments.

The MappyDot Plus will be available on Tindie shortly - https://www.tindie.com/products/blecky/mappydot-a-smarter-micro-lidar-sensor/

The first Litar prototype used some 2D hall effect joysticks (with button click) as a demonstration to enable advanced chord control, vibrato as well as ambidextrous control:

These joysticks were based on the AMS Easypoint N40P107 assembly which uses the AS5013 two-dimensional magnetic position sensor. These are great little compact assemblies, however unfortunately the manufacturer has decided to no longer make these and leave it up to the product designers to make their own magnetic pointer with the position sensor.

This left one of two options, either make your own pointer assembly, or move to a low profile analogue joystick instead. Low profile analogue joysticks with a button click are unfortunately difficult to find it turns out, unless you want large quantities of them (like the ones in the 8bitdo pro controllers):

I set that problem aside for another day and continued to play further with the Litar...

After a while and letting different people play with it, it was found that lateral motion in your fingers on something you are holding onto with the same hand was not only difficult, but actually caused a fair bit of strain after a short amount of use. I tried moving the buttons around in the case to make them more ergonomic, but they were still difficult to play and removed the ambidexterity of the controller.

So I set out to design something different. I wanted to be able to still do vibrato, so I needed a linear sensor of some sort, preferably with some sort of pressure control. Why not a linear hall effect sensor with springs and a simple rectangular button?!

This sort of setup does simplify the chord input a fair bit, but it does still give you varying levels of control input and can still be waggled for vibrato.

Here's a cutaway of what I came up with:

The linear hall effect sensor sits on a PCB, as does two springs on each side of the button. The button moves freely in the hole of the case and is held there by the springs. There's two stoppers on the button part that hits the PCB as it comes down and these stoppers could press a momentary switch. There's a magnet in the center of the button as well.

The sensor array on the Litar has been designed to allow for the lowest latency and best accuracy possible when obtaining a string interaction. Each string has a specific geometry that allows them to be easy to play and makes them large enough to pluck.

The sensors are aligned in such a way that three sensors see a single string in space; two sensors define the string border and a third sensor obtains the string distance. This enures that you don't get miss plucks and it also allows the strings to be packed tighter together over one or two sensor setups (the sensors have a 25 degree field of view). To visualise this, the image below shows the sensor field of views, with the grey areas highlighting the virtual strings:

To obtain the strings, the data from the sensors is brought into an array of sensor distances over time and placed through a rapid-change detection algorithm. The algorithm scans each combination of sensors which create a string and then obtains the correct distance for the string being plucked. It does this by looking at the most recent value as well as its history to determine which string has been plucked, with the latency of a single distance measurement. With this you can also obtain entrance to the string area velocity information, without increasing latency.

And that's how the Litar gets its strings.