So the version I have at this point works just fine, but I was unhappy with how hard it was to attach the Tile Holder Buttons to the 4 pin headers on the PCB. Also the solid core wire that I was using was a little "stiff" and had a tendency to make the buttons tilt a bit. So I made a couple of small changes.  I switched to soft silicone stranded 28 AWG wire and 90 degree headers for connectors.

For the most part the assembly is the same as described previously, but there is one thing to watch out for. 

For switches going on the left side, the connector is installed facing down as in the left picture above. Right hand side button has the connector facing up, and the connector has to be rotated 180 degrees before attaching then to the PCB headers. The wires for for left hand side switches should be 42 mm long and for the right side 46 mm (to allow for the twist in the wires).

Overall it was a bit of work, but totally worth ending up with a much cleaner build.