Close

Log #10: Related work (2)

A project log for lalelu_drums

Gesture controlled percussion based on markerless human-pose estimation with an AI network from a live video.

lars-friedrichLars Friedrich 10/06/2024 at 19:590 Comments

Recently I got to know the project SONIC MOVES that is strongly related to lalelu drums. They present an iOS app that can be used to control music from 3D body tracking on an iPhone XR or newer.

The developer Marc-André Weibezahn has a similar app called Affine Tuning, that seems to be a predecessor of sonic moves. He announced the foundation of sonic moves eight months ago on linkedin.

From the demo videos it appears that the player does not trigger individual notes, but rather controls a path through what they call a 'song blueprint'. The song blueprint consists of MIDI tracks and effect definitions and the player's movements control how the MIDI tracks are played back and set parameters of the effects. Accordingly, the description of Affine Tuning states that it is not meant to be a musical instrument (that you would have to practise), but rather aims to motivate physical exercise.

I like the idea of an intuitive application that does not require any training to use it. I think it can be great fun and very motivating to explore various song blueprints by experimenting with body movements. However, I wonder if the frame rate and latency that can be achieved on a mobile device allows the music output to be controlled precisely enough to play along with other musicians, as it is shown for example in Rapper's delight.

Discussions