Ha, it's always funny to look back at yesterday's problem after having solved it. You feel so dumb for not seeing it right away! It's a bittersweet feeling, but the feeling of solving a hard problem -- even if it was only hard because you're a simpleton -- feels so good. 

Last night, after depleting all of my batteries juice and retiring to relaxation, I realized what was wrong. I was trying to tune for both impulse response and steady state. The problem was, I was trying to put the quad on edge. This is absolutely not helpful. I got confused by the language in the PX4 docs about the rate tuning process, where it was suggested to tilt the vehicle on edge to tune the I gain. I realize now that they were implying while flying the vehicle. Duh. The test fixture was effectively a moment arm on the quad as soon as it rotated about the dowel! After reading many articles on the interwebs, I decided to only test for impulse response (poke the drone) and step response (oscillate with fixed speed between shallow angles). I started by finding the correct P gain (2/3 * frequency_begin_oscillations) and then dialed in D with P and I set to zero. After finding "ballpark" values for P and D. I used the step response to factor in I gain, and dialed P back a bit further...

...Rate controller tuned!

 I am not sure the impact of this overshoot in the step response, and I am also a little bit concerned about the rise time. I was primarily trying to avoid heavy ringing and in some cases minor high frequency oscillations with previous more aggressive tunes I had set up. 

Next was the attitude controller, but that's just P gain so was quite easy to find a good initial value. I did one last "hold in hand" test, and then went to flight test it. It flew! It was actually very smooth, although the attitude control was noticeably sluggish. I haven't implemented the yaw controller yet (been busy, stuff and things) so it is quite difficult to fly. The yaw controller should be quick and easy (rates only), so I am planning on tomorrow being a flight testing day!