This came out of the Game Boy Color work and it applies to every autoregressive port on this project, so it gets its own entry.

The obvious way to check that an optimization did not break the model is to generate a few tokens and compare them against a reference. Three tokens is fast, it fits in a bench ROM, and it feels like enough. It is not enough.

The model picks each token with an argmax over 512 classes, and argmax is a step function. You can inject a large amount of numeric drift into the logits and the winning index does not move, so the output stays byte-identical while the arithmetic underneath is quietly wrong. The test passes. Nothing looks broken.

Two genuinely broken changes sailed through a three-token gate here. First, deleting the live range checks in the fixed-point clamp — those checks demonstrably fire during normal generation, so removing them is a real behavioural change, and three tokens did not care. Second, narrowing the key-value cache entries to int16, when keys in this model peak around 110,944, far outside int16 range. Forty-four overflows occur inside the three-token run itself. Still passed.

A sixteen-token gate catches both, and the reason is structural rather than statistical. Sixteen tokens fills the context window, so errors enter the KV cache and every later position attends over the corrupted entries. The damage compounds instead of being absorbed, and the model collapses into a repeating cycle. That repeating-cycle signature is the same failure the Genesis ROM showed when it started rambling, which is how two ports ended up sharing one diagnosis.

There is a performance version of the same mistake. A three-token bench understates KV and attention work by roughly four to five times, because the position index never exceeds two. Real wins get discarded as noise on that bench and real regressions hide in it.

One instrument turned out to be useless entirely: static instruction counting. It swung from minus seventeen percent to plus one percent on a reorder of lines that changed no semantics whatsoever. Only frame counts under an emulator meant anything, and those were cross-validated across PyBoy and SameBoy, which agree to within poll granularity with tokens exact.

The general rule: a correctness gate has to run long enough for errors to feed back into whatever carries state forward. For a transformer that means filling the context window. Anything shorter is measuring whether argmax is robust, which it is, rather than whether your change is correct, which is the actual question.