Going through a couple redesigns and commissions later, I've decided to make a modular device, multiplexing rotary encoders, buttons and whatever else, made with a configurator (wip) to generate the setup for the teensy to read (might just use serial + the eeprom instead of reprogramming..)
Cool screen print on the board