I had trouble with the keyboard code so I went back to working on the mouse part. I got it working almost the way I want it with help from someone on reddit. The mouse works after playing around with the sensitivity value in the code. At first the pointer moved really slow. I lowered the value to around 25 and it moves fast enough but now the pointer wanders a little. The left and right click buttons are working.

So back to the keyboard, I have worked out the pin mapping for the keyboard to teensy connections and I did some work on the matrix and the different states (shift, fn..) for the button presses. These are the pin mappings

columns: keyboard pin   |   teensy pin (with arduino pin numbers)
                     1  |  27
                     2  |  0
                     3  |  1
                     4  |  2
                     5  |  3
                     6  |  4
                     7  |  5
                     8  |  7
                     9  |  8
                    10  |  9
                    11  |  10
                    12  |  11
                    13  |  12
                    14  |  13

rows:    keyboard pin   |   teensy pin
                     A  |  22
                     B  |  21
                     C  |  20
                     D  |  19
                     E  |  18
                     F  |  38
                     G  |  39
                     H  |  40
                     I  |  41

Hopefully that makes sense. And here is the matrix. I don't know how to paste it without it getting all jumbled so I'll post a screenshot.

I will continue to work on this and hopefully learn how to get it working.