After a bit of fiddly soldering, I've attached a Pico W to the Sony's circuit board. The connections are very simple. The two battery contacts on the circuit board - plus and minus - are connected to VSYS and GND on the Pico. The Sony's "refurbished" battery pack is made up of 4 AA rechargeables, outputting 4.8V in their prime, which falls well within the 1.8-5.5V range accepted by VSYS. Thus, I can power the whole contraption from the original battery pack with no visible external changes to the phone.

Pin GP15 is connected to the little speaker which would have originally acted as the ringer, along with another GND. Using CircuitPython, I can program the Pico W to play a low-fidelity MP3 via the ringer whenever I feel like it (e.g. when I receive an email).

TLDR - I've basically turned the obsolete analogue phone into a functional basic tone pager, powered by a Pico W.