I found my rubbish speaker...   I found a microphone...  I found a good speaker...

The goal is to play a sound with the good speaker.  Then play one with the rubbish speaker.  Then figure out the main differences in the sound via doing an FFT and looking for the most major differences.

Next, calculate a signal which will cancel the differences, and repeat the whole process with the new signal added in for the trash speaker only.   Over time, the differences will diminish towards zero.