Here's where it gets weird. We wanted a large language model to actually play the game running on the board, so we gave the firmware a small control channel: a WebSocket /agent endpoint that runs the game in lockstep. You send one action, the game advances a few tics, and you get back a structured JSON observation — then the game pauses until the next action. That determinism is the whole point: the model always reasons about a frozen, coherent frame instead of a scene that keeps moving while it thinks. It's also why, in the video, the marine moves in little bursts — the game is literally paused between the model's moves.

On top of that we wrote a tiny MCP server. MCP (Model Context Protocol) lets any MCP-capable client — Claude Code, for instance — register external tools; ours exposes the game as observe, move_forward, move_back, turn_left, turn_right, strafe_left, strafe_right, fire, use, select_weapon, and get_map. Point the client at the board's IP, and the model can just play — no glue code, no screen scraping.