Close

The tok/s counter was lying, and how I caught it

A project log for Transformers on Retro Game Consoles

Real transformer language models running on stock NES, SNES, Game Boy Color, Genesis and N64 hardware, measured in actual cycles.

scott-boudreauxScott Boudreaux 6 hours ago0 Comments

The N64 port shipped with a token rate on screen, counting up live as the text generated. About 60 tokens per second. It is in the video. That number was wrong by roughly a factor of fifty. The real figure is 1.23 tokens per second.

Here is what gave it away, eventually. The counter read essentially the same on the scalar build and on the RSP overlay build. It should not have — those two differ by a real 1.8x. A gauge that does not move when the thing it measures moves is not a gauge.

The cause was that the rate came from a cycle-count read that does not advance against real time under emulation. Under ares this workload runs at roughly 0.35 to 0.63 times real time, drifting with load on the host GPU, so a rate computed that way tracks the host machine rather than the console.

Fixing it meant two independent instruments that had to agree before I believed either. CP0 cycle counts, and counting vblanks. Vblanks are the honest clock here, because the console produces sixty of them a second no matter what the emulator is doing underneath. Both now put the scalar build at 1.23 tok/s and the RSP overlay at 2.19 — a 1.78x win for the RSP, and the two methods agree to within a tenth of a percent.

A 4.769x figure also circulated for this port. That one was a ratio of CP0 cycle counts taken on an int8 blob, which is not what ships. The shipped blob is ternary, and on ternary the RSP margin drops to 1.78x, because ternary has already eliminated most of the multiply work the RSP was winning on. Two different builds, two different quantizations, one number quoted across both.

The uncomfortable part is the part worth writing down. A broken counter made this port look roughly fifty times better than it was, and it sat there for weeks without anyone questioning it, myself very much included. Nobody audits a number that flatters them. If the counter had read 0.02 tok/s I would have found the bug that afternoon. That asymmetry is the real defect, and it is not in the code.

Still outstanding: none of this has run on N64 silicon. Every figure above is ares. If you own the cartridge and the console, I would much rather be corrected than quoted.

Discussions