after some suggestions from bristol hackspace mailing list, I traced the noise from the sensors on the router table (end stops). The opto sensor was put down the same cable, so the noise was transferred to its signal too. I separated the opto out onto its own shielded cable and the noise signal fell to about 600mV peak to peak. This fixed the RPM reading and now the whole system is working!