It turns out someone did something similar (but in a much more polished way) and turned it into a product :

https://arstechnica.com/gaming/2021/06/playdate-the-console-with-a-crank-gets-july-preorder-for-179-game-details/

And even before that, I found this project : https://tomtilley.net/projects/hipster-controller/

Sometimes it's nice to be the first to do something, and sometimes, when you do something weird, it's reassuring to see you're not the only one with strange ideas ;-)