Iterated on the shaft coupler trying to eliminate a problem which was instead the result of the knurling I had put on the AL shaft itself.  I had just pressed them in the jaws of my linesman pliers to generate the knurling, which made them wobbly.

I was really unlucky in my initial tests and had randomly lined up the error with the flat on the shaft multiple times. When I flipped the shaft around to the smooth end, the wobble went away.  This was basically a waste of time.  

They don't like to come off once pressed on so many of them broke.