As explained in the time server document, you can get the NTP server to keep statistics on the offset and later compute an adjustment to the fudge line in ntp.conf. The suggested computation had to be modified, probably because I'm running ntpsec instead of ntp. This was what I ended up using:

awk '/SHM\(0\)/ { sum += $5 * 1000; cnt++; } END { print sum / cnt; }' < /var/log/ntpsec/peerstats

The original used a search filter of 127.127.28.0 but it seems that ntpsec records GPS readings as SHM(0). Also the output is in milliseconds so has to be translated to seconds for adjusting the time1 value on the fudge line. This is what I have currently.

fudge 127.127.28.0 time1 0.118 refid GPS

a difference of 2 milliseconds from the previous value. Here's a typical output of ntpq -p:

     remote           refid      st t when poll reach   delay   offset   jitter
===============================================================================
 0.debian.pool.n .POOL.          16 p    -  256    0   0.0000   0.0000   0.0019
 1.debian.pool.n .POOL.          16 p    -  256    0   0.0000   0.0000   0.0019
 2.debian.pool.n .POOL.          16 p    -  256    0   0.0000   0.0000   0.0019
 3.debian.pool.n .POOL.          16 p    -  256    0   0.0000   0.0000   0.0019
 192.168.16.1    .INIT.          16 u    - 1024    0   0.0000   0.0000   0.0019
*SHM(0)          .GPS.            0 l   49   64  377   0.0000  -1.1543   0.6951
+time.cloudflare 10.26.8.188      3 u   42   64  377  15.2933   4.4961   1.1248
+time.cloudflare 10.26.8.188      3 u   50   64  377  15.2943   5.6666   0.5611
-220-158-215-20. 202.46.178.18    2 u   17  128  377  39.1423  -6.5729   0.8229
-cosima.470n.act 203.35.83.242    2 u   15  128  377  18.9447   4.4120   1.0238

I may fine tune it further after running it for longer, but at this point we are in millisecond territory which is far more accuracy than I need for home purposes. There's no point tweaking it much more given the jitter in the absence of an accurate PPS hardware signal which is coupled into the kernel.

Next task is to register the RPi as a time server for my workhorse PC.