EyeBREAK (Morse Blink BLE Keyboard)

I have published the code at https://github.com/m-bw/EyeBREAK; it includes the training notebook as well as the firmware.

I was testing if a 36x36 image input would work, and it in fact does achieve the same or better accuracy (97%) with a third of the parameters (10k) and also halves the inference time (20ms). This brings the FPS back up to around 28.

I also moved to esp-nimble-cpp for the BLE stack and it does indeed reduce flash usage by about .2MB.

I finally got BLE keyboard functionality working. Although there is an example provided with ESP-IDF, getting the HTTP server (for the camera stream), WiFi, and Bluetooth stacks running at the same time is a bit finicky.

The main issue that arose was IRAM size limitations. I tried the config options found in this IDF doc section, but evidently there still wasn't enough room. There is a new config option found in the ESP-IDF master branch that allows some DRAM sections to be used for IRAM, so I switched to master instead and enabled that option. Then, DRAM wasn't big enough, so I had to force Bluetooth and WiFi to allocate from PSRAM instead, among other things, which finally made it work.

The BLE HID example uses Bluedroid, which is larger than the alternative NimBLE stack. Perhaps switching to this could reduce these issues? I've actually used NimBLE with Mynewt many years ago for another BLE HID project, so it might not be too difficult.

As for the functionality itself, the keyboard supports typing characters as well as backspace, by holding the eye closed for a longer amount of time. Also, to provide visual feedback, the individual Morse dots/dashes are typed out while a character is in progress, then erased once it is finalized and replaced by the final character. I'll try to show this in a video recording.

I trained a very small CNN (30k parameters, 4 layers) on the dataset and achieved higher accuracy (96%). It also works better in the wild, even without the Haar cascade preprocessing.

Through TFLite for Microcontrollers, I was able to get it to run on-device. With 8-bit quantization, it runs at ~18 fps, which is pretty good.

The whole train-deploy pipeline is a bit of a mess though, since I used Pytorch:

Pytorch (Lightning) -> ONNX -> TF -> TFLite -> TFLite (Quantized).

Will add another demo soon.