Got the clearance right to hold the magnets in place and minimum wobble in the shaft.  It's going to be balancing act between getting magnets as close as possible to the PCB and still minimize/prevent rubbing.  I ordered a shaft assortment and a few more bearings.  Can drill out the 5mm hole to about 10mm without impacting the traces.  PCBs have shipped but a few more days to wait until they get delivered.