Despite all the problems, I'm persevering. Today the common-cathode bi-color matrices arrived, and I was able to try them with the HT16K33:

Of course, the pattern is pretty much random, because I still need to adjust the software for the pretty much random column and row assignment I made. But you can see the colors, and they are nice (the slight variations of color on the photo are due to the camera sensor interfering with the PWM).