Ok, just cut the traces from the TX/RX pins and rewired from 2 pins that were being used for the RGB button. Haven't got a working multimeter so will have to check in a few days but this should allow the serial chip to sleep.

For anybody using a nrf52 feather just STAY OFF TX/RX pins, they behave unpredictably and they'll prevent your feather from going to sleep.

Uploaded V4 board that has the fix. If anybody does get one made, let me know how it goes. If anybody wants a v3 board, happy to ship them over, still have a few spare. You'll need to cut 2 traces and solder 2 wires like in the image above.