So, I linked it up to puredata...

End of last year, I was pointed to a software called puredata, that can be used to synthesize sounds, play sound files, serve as a music sequencer and so on. In this log entry, I show that the gesture detection backend of lalelu drums can control a puredata patch via open sound control (OSC) messages.

The screencast shows the desktop of my Windows development PC. The live video on the left is rendered by the frontend running as a python program on the PC. On the right, a VNC remote desktop connection shows the puredata user interface, running on a raspberry pi 4.

The backend sends two different types of OSC messages to the puredata patch via cable based ethernet. The first type of messages encodes the vertical position of the left wrist of the player. It is used by puredata to control the frequency of a theremin-like sound. The second type of messages encodes triggers created from movements of the right wrist and elbow. The backend uses a 1D convolutional network to create this data, as it was shown in the last log entry. In puredata, these messages trigger a base drum sound.

Here is the output of htop on the raspi4. It can be seen that there is significant load, but there is still some headroom.

Thanks go to Matthias Lang, who gave me the link to puredata. The melody in the video is taken from Borodin, Polovtsian Dances. The basedrum sound generation was taken from this video.