In the last episode, after some fruitless searching:

So at this point, it would be great if some better understanding of constrained-nut-on-spinning-screw dynamics happened to fall from the sky.

Today while following CNC on the Desktop Hack Chat and pondering what question I could ask that group, I found Orang Vahid Araghi's thesis from 2009: Friction-induced Vibration in Lead Screw Systems.

His introduction includes:

Disproportionate to the popularity of lead screws and their wide range of applications, very little attention has been paid to their dynamical behavior. Only a few works can be found in the literature that touch on the subject of lead screw dynamics and the instabilities caused by friction.

And regarding the first of two real-world examples discussed:

... an extra force applied ... in the direction of motion causes the system to generate audible noise, ...

This sounds like a) Orang couldn't find anything either, and b) he had to solve something like my problem.

His solution involves active motor controllers, which is probably out of scope for this project. But maybe there is enough clue in here to tell how to avoid triggering the "instability". At 277 pages, this may take some digging...