Transformers on Retro Game Consoles

This came out of the Game Boy Color work and it applies to every autoregressive port on this project, so it gets its own entry.

The obvious way to check that an optimization did not break the model is to generate a few tokens and compare them against a reference. Three tokens is fast, it fits in a bench ROM, and it feels like enough. It is not enough.

The model picks each token with an argmax over 512 classes, and argmax is a step function. You can inject a large amount of numeric drift into the logits and the winning index does not move, so the output stays byte-identical while the arithmetic underneath is quietly wrong. The test passes. Nothing looks broken.

Two genuinely broken changes sailed through a three-token gate here. First, deleting the live range checks in the fixed-point clamp — those checks demonstrably fire during normal generation, so removing them is a real behavioural change, and three tokens did not care. Second, narrowing the key-value cache entries to int16, when keys in this model peak around 110,944, far outside int16 range. Forty-four overflows occur inside the three-token run itself. Still passed.

A sixteen-token gate catches both, and the reason is structural rather than statistical. Sixteen tokens fills the context window, so errors enter the KV cache and every later position attends over the corrupted entries. The damage compounds instead of being absorbed, and the model collapses into a repeating cycle. That repeating-cycle signature is the same failure the Genesis ROM showed when it started rambling, which is how two ports ended up sharing one diagnosis.

There is a performance version of the same mistake. A three-token bench understates KV and attention work by roughly four to five times, because the position index never exceeds two. Real wins get discarded as noise on that bench and real regressions hide in it.

One instrument turned out to be useless entirely: static instruction counting. It swung from minus seventeen percent to plus one percent on a reorder of lines that changed no semantics whatsoever. Only frame counts under an emulator meant anything, and those were cross-validated across PyBoy and SameBoy, which agree to within poll granularity with tokens exact.

The general rule: a correctness gate has to run long enough for errors to feed back into whatever carries state forward. For a transformer that means filling the context window. Anything shorter is measuring whether argmax is robust, which it is, rather than whether your change is correct, which is the actual question.

The N64 port shipped with a token rate on screen, counting up live as the text generated. About 60 tokens per second. It is in the video. That number was wrong by roughly a factor of fifty. The real figure is 1.23 tokens per second.

Here is what gave it away, eventually. The counter read essentially the same on the scalar build and on the RSP overlay build. It should not have — those two differ by a real 1.8x. A gauge that does not move when the thing it measures moves is not a gauge.

The cause was that the rate came from a cycle-count read that does not advance against real time under emulation. Under ares this workload runs at roughly 0.35 to 0.63 times real time, drifting with load on the host GPU, so a rate computed that way tracks the host machine rather than the console.

Fixing it meant two independent instruments that had to agree before I believed either. CP0 cycle counts, and counting vblanks. Vblanks are the honest clock here, because the console produces sixty of them a second no matter what the emulator is doing underneath. Both now put the scalar build at 1.23 tok/s and the RSP overlay at 2.19 — a 1.78x win for the RSP, and the two methods agree to within a tenth of a percent.

A 4.769x figure also circulated for this port. That one was a ratio of CP0 cycle counts taken on an int8 blob, which is not what ships. The shipped blob is ternary, and on ternary the RSP margin drops to 1.78x, because ternary has already eliminated most of the multiply work the RSP was winning on. Two different builds, two different quantizations, one number quoted across both.

The uncomfortable part is the part worth writing down. A broken counter made this port look roughly fifty times better than it was, and it sat there for weeks without anyone questioning it, myself very much included. Nobody audits a number that flatters them. If the counter had read 0.02 tok/s I would have found the bug that afternoon. That asymmetry is the real defect, and it is not in the code.

Still outstanding: none of this has run on N64 silicon. Every figure above is ares. If you own the cartridge and the console, I would much rather be corrected than quoted.