A proper schematic helped -- the connections are now all correct. Of course I forgot to move the pins for the matrix closer together, so the pin spacing is wrong now...

But I think that I finally ran out of mistakes to make. Ordered the fixed PCBs, and prepared a 4x panelized PCB fitting in 5x5cm, for a "production" order. I will, however, wait until the current version arrives and tests before I order those.