After ordering more than a dozen connectors and pins off of Digikey, I found a connector that is just what I am looking for, the TE 2-173983 series connectors. I got the -7 at Digikey, but any of the series has the same contacts. The key is that the board mate is not a solid loop, but two separate fingers somewhat like a rock climbing carabiner. This gives it some spring on the board mount section, perfect for this project. That said, the top half of the contact is all wrong. It is designed for a wire press fit, so I'll need to do some hacking.

It took more than 10 hours of searching to find this one, but excited about the results.