Here are the technical details for the smarties here on Hackaday.

Google TensorFlow is an AI framework and google recently released MoveNet - an incredibly fast and accurate skeletal tracking model they trained (called a "pose estimator" in the research world). You can try out a live demo of MoveNet here. Thanks to TensorFlowJS this can be made to run in-browser. I got that working and then added some code to detect various gestures like punching, blocking, and dodging. The gesture recognition is done by simple heuristics. For example, "dodge" is activated when both wrists move above the nose. There is hysteresis to prevent "dodge" from being triggered multiple times from one movement.

Next the challenge was to get Punch-Out!! to work inside a browser. I had been using fceux as my NES emulator, and I was able to use a cool tool called emscripten to compile fceux into javascript so that it would run inside a browser. 

Now comes the delicate issue of ROMs. The game's binary, normally stored on the cartridge, is called a ROM (read only memory). The only legal way I'm aware of to obtain ROMs is to copy them off a cartridge you own, and from there on out you can never play both the ROM and the cartridge at the same time or you will be violating Nintendo's copyright. Copying a ROM off a cartridge is not a simple task. It's not like the cartridge plugs into your computer after all. What hackers have done is create a circuit called USB Copy NES that can be inserted between the Nintendo's motherboard and chip, which allows you to plug in a USB cable and then copy the ROM to your computer using software called USB Copy NES Blue.

So far these were straightforward steps and not terribly difficult to a hacker. What comes next though is a... suitable challenge. Because it takes longer to punch up into the air than it does to press a controller button, the game was unplayable because the opponents moved faster than you could react. The only way I saw to fix this is to hack the game and slow the characters down. However these games aren't open source. Nintendo doesn't offer public github repos after all. So I had to disassemble the ROM. If you're not familiar, disassembly means taking the compiled game (a binary file) and reverse-engineering the original source code. It is crazy hard. Luckily I had help from a few developers in the retro gaming world to get me started, and after 2 months I was able to finish disassembling enough of the game so that I could slow each character down and make it a fair fight.

When you play, it's super fun and it feels like you're really INSIDE the game. That gave me the idea that something should happen to me when my character gets hit in the game. Something like electric shocks. I plugged an AdaFruit Feather 32u4 RFM69HCW (arduino with a radio transceiver, also called a "radiofruit") into my computer and used that as the transmitter. I wrote some javascript that uses webusb to instruct this radiofruit to transmit a "shock" command over the air anytime my character gets hit. The receiver is mounted into an armband, and uses a relay to power a high-voltage shocking circuit stolen from a shocking hand buzzer - so it's a good jolt, but nothing that will really harm you. The shocks are delivered using two copper pads mounted on the armband (a typical smartphone armband for when you're running). If you watch the video, you can hear me getting shocked and screaming like a cartoon character. What fun!

Enjoy :-)