I did some experiments with painting keycaps. I painted the letters of the printed caps (which are recessed in the keys) with acrylic color (left), then sanded the cap smooth the  next day (right). The result is not perfect as there are small imperfections in the print where the paint still sticks, but its good enough for now.