About 14 years ago I set upon making the smallest PONG arcade in the world, after scavenging a tiny 5x1 HD44780 display from a server I found in e-waste. Unfortunately that display put up a fight and got cracked. It still worked, so I pushed on. When I had everything finished, it failed...

(and even if I ever stumbled across another one of those displays, your project easily beats mine size-wise, being a little under half the size)

Long story short - instead of using potentiometers to control the paddles, I measured skin resistance and developed some kind of windowing algorithm to always keep the input ranges within a window for the paddle movement.

Maybe that helps you on designing inputs for this thing. Just put two traces on either side of the display for the players to press their fingers against, measure resistance, perform some averaging and use that to control the paddles.