The ATMega8 falls short 1 data line to scan all the matrix rows and columns. I tried different solutions using a diode to do bidirectional scanning (+5V/high impedance/GND) for one of the data lines. Parasitic capacitance, non-zero resistance of the button (it's about 80 ohms IIRC) and the need for both a pull-up and a pull-down resistor on the pin, combined with the diode were just too much: I read nothing. Without the diode it kind of worked, but then only unidirectional. And I had to do some pin toggling and wait not too long for the parasitic capacitors to (dis)charge or I would read something unexpected...

So I need to disable the Reset pin. That's a bit scary. To see if I'll still be able to update the bootloader, and to make entering the bootloader a little more convenient, I decided to try the 'make update' functionality of USBAspLoader, and the change I made was that I can now enter bootloader by pressing the "P" button when inserting the keyboard. And it works! :)

See the small change in my own fork: https://github.com/zeekoe/USBaspLoader/commit/b4a26c5f4f4fb5a620630d5aa0ba0e0990053c55

Sadly, USBAspLoader can't seem to set fuses (that might be something only an external programmer can), so I need to connect it for one time to disable the reset line.

Alongside I found the missing UpArrow,PageUp,LeftArrow on the keyboard, so now is my matrix complete!