I added load sharing to the charger so that the device can run off USB (or any other external 5V supply) or off the battery. Either external or internal power are then fed into a pololu 5V step up/down regulator. Here's a picture of the charger circuit and the explorad board:

In the above picture, the battery is supplying power (the red wire from the Teensy is open, upper left corner). After a while (1h 15m), I connected the charger to the Teensy to give it 5V from USB, so it switched to that supply line and started charging:

It stopped charging after about 6h, but the device was still supplied from USB. So I disconnectd USB power again (6h 15m) and the battery was used again. The was no power loss in between, so I can connect or disconnect external power while the device is running.

I have no idea why the readings were so flaky, but it might be due to the very old battery. I'll see if I have a newer one somewhere.