First pass mimicked VNC's approach to updating the screen: a frame is a list of rectangles, each encoded on its own, unchanged parts of the screen simply not sent. VNC's Tight encoder picks whatever makes the smallest payload, so that's what I did too. Deflate the flat bits, JPEG the busy bits.

It was slow. So I wrote caps_test.py, ran it on the actual board, and got these:

zlib inflate on this firmware is about seven times more expensive per pixel than a JPEG decode. Picking the smallest payload was spending 61 ms per screen to save bytes on a link that was 5% utilised.

So the encoder stopped asking "which is smallest" and started asking "which one does the device finish first". Every rectangle gets priced in client milliseconds: decode, plus the time for the bytes to arrive, plus a cost per UDP datagram because every recv in MicroPython costs something too.

def cost_ms(self, enc, pixels, nbytes):    if enc == P.ENC_SOLID:        decode = self.solid_ms    elif enc == P.ENC_JPEG:        decode = self.fixed_ms + pixels * self.jpeg_us_px / 1000.0    elif enc == P.ENC_DEFLATE565:        decode = self.fixed_ms + pixels * self.deflate_us_px / 1000.0    else:        decode = self.fixed_ms + pixels * self.raw_us_px / 1000.0    frags = max(1, (nbytes + self.frag_payload - 1) // self.frag_payload)    return decode + nbytes * self.ms_per_byte + frags * self.frag_ms

JPEG wins almost everywhere now. Deflate stays in because a device without a JPEG decoder can still use it, and the client says what it can decode in its HELLO packet, so the server just routes around whatever is missing.

Dirty rectangles are a numpy one-liner on 32x32 tiles. If more than 60% of the tiles changed, it's cheaper to send the whole frame than eight small ones:

diff = np.any(cur != ref, axis=2)
tiles = diff.reshape(th, t, tw, t).any(axis=(1, 3))
if tiles.mean() >= self.full_frame_ratio:    return [(0, 0, self.width, self.height)]

Streaming DOOM E1M1 at 128x128 and 20 fps comes out at about 2,146 bytes per frame while moving, 42 kB/s. The link was never the bottleneck.