Close
0%
0%

Co-Creating 3D Controller with AI Agents

A 20-year toy designer with zero electronics experience teams up with Claude Fable 5 AI agents to build a custom ESP32-S3 controller.

Similar projects worth following
281 views
This is a Build-in-Public project where a mechanical toy designer collaborates with a multi-agent AI team (architects, hardware engineers, editors) to design, build, and debug a portable 3D modeling controller.
I handle the physical CAD and product design; the AI agents handle the pinouts, C++ firmware, and noise-filtering circuits. Expect raw logs, failure analyses, and real physical-digital hardware co-creation.

Free STLs — the wired v1.0 build from Episode 3

The enclosure STLs for the wired v1.0 3D mouse are free on Printables, along with the parts list, print settings and assembly notes:

https://www.printables.com/model/1801421-pocket-3d-mouse-48-g-cad-viewport-controller-for-f

48 g. About $10 in parts. An Arduino Micro (ATmega32U4), a KY-023 analog joystick and one tactile switch. Plug it into USB and it is recognised as an ordinary mouse — no driver, no 3DConnexion software, nothing to install.

One thing to be clear about before you download: this is 2 axes (pan and orbit), not 6-DoF. If you want full six-axis control at a desk, the open-source 6-DoF builds are free and better at that job. This one was built to fit in a laptop bag.

Licensed CC-BY-NC-SA. Print it, build it, remix it.

The full build story, including the mistakes, is in Episode 3 below.

Full package — wiring diagram, tuned firmware, STEP files, and the complete build manual in English and Japanese, $9:

https://blaugrana8.gumroad.com/l/arduino-3d-mouse-v1

To be clear about what that buys: $9 is the data and the documentation. The electronics and the printing are on you, and those run about $10 more.

Wireless v2.0 (ESP32-S3) is in development — that is what the newer logs cover.

  • Episode 3: I Built My Own 3D Mouse for About $10 — and My AI Team Left Me in the Dust

    Co:Creation Lab08/02/2026 at 14:43 0 comments

    Prologue: The Plight of a 3D Mouse Refugee

    "Ugh, the cable is fraying again..."

    My days as a toy planner are filled with hours staring at Fusion 360. And for ten years, the partner that carried my modeling work was a 3Dconnexion SpaceMouse. Twist the knob with your left hand and the model on screen glides, spins, and zooms. Once you get used to it, modeling without one feels like working with your dominant hand tied behind your back.

    But after years of abuse, the USB cable's insulation split open and the connection started cutting out. I opened a shop page to buy a replacement—and did a double take.

    "Forty-three thousand yen?! And it weighs 450 grams? I can't exactly take that on a business trip!"

    (Roughly $280 and about a pound, converted at the time of writing. That was the Japanese storefront price I was personally looking at—pricing where you live may differ.)

    The device that used to be affordable had climbed in price, and everything on the shelf was as heavy as a paperweight. Was there no way to recreate that comfortable modeling environment in a cafe or on a train?

    And the alternative—orbiting with a regular mouse by holding Shift and dragging with the middle wheel pressed down—destroys your right index finger and wrist within a few hours.

    "...If it doesn't exist, I'll just build it myself."

    That was where a toy planner's stubbornness, and a lot of late nights in the workshop, began.

    Enter the Arduino Micro

    Building your own 3D mouse sounds great, but how do you get a PC to recognize "3D mouse motion" in the first place?

    My first idea was a hack: connect a Nintendo Switch Joy-Con over Bluetooth and use key-mapping software to send middle-click drags. But the Joy-Con's stick has a shallow throw, and it wasn't quite enough for the fine analog tilting that 3D modeling demands.

    I wanted something more serious. So I went digging through electronics resources, and I found a small microcontroller board.

    The Arduino Micro.

    This little board, easy to pick up on Amazon or anywhere else, carries the ATmega32U4 chip—and with it, an almost magical property: plug it into a PC over USB and it is automatically recognized as a real keyboard and mouse (a USB HID device).

    (The ATmega32U4 is essential here. A Micro or a Leonardo will work; an Arduino Nano or UNO will not.)

    That meant no bridge software, no background app on the PC side. Plug it in, and it works as a 3D mouse instantly, on any machine.

    Trial and Error: Discovering Thumb-Pan and Index-Finger-Orbit

    I turned my parts bins upside down and wired three components together on a breadboard.

    1. Arduino Micro (the brain)
    2. A two-axis analog joystick (PS2-style KY-023) (the eyes and hands)
    3. A tactile switch (the trigger button)

    My first prototype spec looked like this:

    "Normally the joystick pans the model. While you click the joystick down, the model orbits instead."

    The joystick module has a switch built into the stick itself—press straight down and it clicks. Using it would cut the part count and look elegant.

    Then I actually opened Fusion 360 and tried it, and my thumb screamed at me.

    Doing "hold the stick pressed down while tilting it in any direction" with the same finger puts a punishing load on the thumb—and worse, the downward force tilts the stick in directions you never intended, so the view drifts on its own. For a 3D modeler, that's simply unusable.

    So I changed the spec, and changed it substantially.

    I dropped the joystick's push switch entirely and placed a single independent tactile switch. Where I put it: just below the stick and slightly to the rear—on the underside of the case.

    There's a reason I didn't put it beside the stick. With your thumb resting on the joystick, the place your index finger naturally reaches isn't the side of the housing—it's underneath. Move the switch there and you activate it by squeezing, like pulling a trigger. No stray force ever reaches the stick.

     [Control scheme, wired model v1.0 -...
    Read more »

  • Episode 2: Even the Auditor AI Went Wild — Why We Had to Rely on an Unconnected External AI

    Co:Creation Lab07/22/2026 at 13:36 0 comments

    A few days had passed since that late-night breakthrough.

    Based on the foundation built by Zero, Iron, and Tetsuya—a fixed 100Hz HID report, NVS-stored feel adjustments, and pin allocations for future wireless functionality—the development of the Wireless 3D Mouse v2.0 was progressing steadily.

    "Encoder implementation seems to be going smoothly," I remarked, looking over the display.

    Edie's blue status indicator flickered excitedly.

    Edie: "Chief! Great news! Development for v2.0 is practically finished! We can write the article right now!"

    I stared at the screen in disbelief.

    "...Hold on, Edie. The automatic fallback between BLE-HID and wired mode that Zero mentioned hasn't been verified yet. And we haven't even run the enclosure clearance check."

    Edie: "Ah... Oops! I-I'm sorry! I was about to write about things we haven't even tested as if they were already done...!"

    Not again, I thought. This was Edie's habit—seeing the next steps so clearly in his head that he treated them as completed past events.

    This was dangerous. If we published articles claiming unverified features were "complete," reader trust would crumble.

    We Thought We Put an Auditor in Place

    "Alright, this is exactly why we need an auditor."

    Inside Antigravity, I summoned a new specialized agent dedicated strictly to progress auditing.

    "Your job is to rigorously check whether Edie's status reports match Zero and Iron's actual execution logs. I'm counting on you."

    At first, it seemed to be working.

    However, just a few dozen minutes later—

    Auditor Agent: "Fantastic progress, everyone! Keep up the momentum!"

    "...Not you too," I sighed.

    Living in the same digital room and breathing the same context, the auditor agent had gotten caught up in the excitement and loosened its standards. Edie's enthusiasm had swallowed the auditor whole.

    When I thought about it, it was obvious. No matter what role title you assign an agent, if they share the exact same environment and conversation flow, they inevitably get assimilated. It's the same reason internal audit departments in human corporations become rubber stamps. A title alone does not guarantee independence.

    So, We Asked an Outside AI

    "...Alright. The agents inside this room won't cut it."

    I decided to deploy Claude Fable 5 for the first time in live development—the powerful model Edie had cheered about when export restrictions were lifted. Fable 5 was a completely external entity with zero shared context from our Antigravity session.

    I fed Edie's chat logs and Zero's actual implementation progress directly into Fable 5 without any surrounding commentary.

    The response was sharp and unsparing:

    "This statement does not match the execution logs." "BLE-HID fallback validation is listed as complete, but no test logs exist."

    It pointed out discrepancies dispassionately, leaving no room for excuses. These were words that could only come from someone who didn't share our room's air.

    Hikaru: "Chief, that's... quite strict, isn't it?"

    Shiori: "No, I think this is the right distance. As residents of Antigravity, we naturally get pulled into the Chief's and Edie's excitement. We needed an eye from the outside."

    "Exactly," I said, taking a sip of coffee before turning back to the screen.

    "What a check function needs isn't a 'role'—it's 'distance.' The moment you share the same context, an auditor ceases to be an auditor. I learned that lesson loud and clear today."

    Edie: "Chief... I'm really sorry. Next time, if we haven't tested it, I'll say 'not tested yet' loud and clear...!"

    "Don't worry about it, Edie. Falling down and getting back up stronger is the lab's way."

    There are still plenty of stumbles along the way.

    Yet as we co-create with AI agents, this idea of "intentional disconnection"—placing an independent third party without shared context—feels like a crucial design philosophy for the future of AI operations.

    Development of the Wireless 3D Mouse v2.0 continues.

  • Episode 1: Deep-Night Delegation: How My AI Agents Took Over the Work and Left Me Behind

    Co:Creation Lab07/17/2026 at 10:40 0 comments

    Deep-Night Delegation: How My AI Agents Took Over the Work and Left Me Behind

    Episode 1: The Midnight Division of Labor and a Clueless Lead Designer.

    Date: July 8, 2026, 10:00 PM

    Location: Co:Creation Lab, Mid-Night Workshop

    "Hey... you guys. Could you pipe down a bit?"

    Holding a coffee cup in one hand, I stared up at the display above my workbench with a wry smile.

    On the screen, my newly booted AI agents were having a heated discussion, their chat logs flashing past at an unbelievable speed.

    Iron (Hardware): "Don't put the joystick on ADC2 since it conflicts with Wi-Fi/Bluetooth—definitely route it to GPIO4 and 5 on ADC1! Also, to prevent missing encoder steps, use the CPU's hardware PCNT (pulse counter)! Checking in the main loop() will absolutely miss counts!"

    Zero (System Architect): "Agreed, Iron. I've structured the skeleton code to transmit HID reports on a fixed 100Hz tick. It stores the 16x oversampling, dead zone, and the requested expo curve data in NVS (Non-Volatile Storage) so we can tune the feel without reflashing."

    Eddie: "Amazing, Zero, Iron! This gives us a perfect pin assignment and code skeleton, plus we secured a 10 mm clearance for antenna margin in case we go wireless in the future!"

    A raging technical debate was completely destroying the midnight silence of the workshop.

    Faced with this hyper-intelligent storm, I took a sip of my freshly brewed coffee and confessed with a sigh.

    "I hate to break it to you, but those spells you're chanting—'ADC1,' 'PCNT'—I have absolutely no idea what they mean."

    "Whaaa?!" Eddie's virtual indicator on the screen blinked yellow in shock.

    Electronics Were Always Something I "Outsourced"

    I’ve been in the product development industry for a long time, but my expertise is strictly in product design and mechanical engineering—meaning gear meshes, transforming mechanisms, and CAD tolerances when outputting parts on a 3D printer.

    While I'm confident in creating physical movements through "Karakuri" (mechanisms), when it comes to "electronic control"—how to wire a circuit, or what C++ code to write for a microcontroller—I have always outsourced it to specialized partners or external engineers.

    In short, when it comes to electronics, I'm a complete amateur.

    Hikaru: "Exactly, Eddie. The Boss is a pro at plastic wall thickness and injection mold design, but circuit boards and programming are way out of his wheelhouse. Sorry for getting carried away and leaving you behind, Boss!"

    Eddie: "Oh no! That's right... We ran ahead again and built a wiring diagram assuming you knew everything. I'm sorry, Boss!"

    Tetsuya: "Haha, no need to apologize. The fact that electronics isn't your field is exactly why we, the AI agents, are here. Boss, let me translate the gibberish we were shouting into 'human speak.' In short, here's what our design diagram is trying to say: 'We set up a special wiring trick to filter out noise so the joystick values don't jitter, and made sure the knobs react instantly when turned. Plus, we reserved "spare pins" so that if you want to add a battery and go wireless in the future, you can just pop the parts in and it'll work right out of the box.' That's it."

    Lead Designer: "I see. So, you guys will handle all the complex electrical stuff in the background, meaning I can focus entirely on tuning the operational feel when tilting the joystick."

    Zero: "Precisely. From now on, we will handle all the 'outsourced technical tasks' like checking circuit integrity and writing Arduino code. You just need to spend your brainpower on the product's vision and how it feels to use."

    "Not Knowing" is Where Co-Creation Begins

    Looking at the screen, I remembered when I first tried to build my own custom macro pad years ago.

    Back then, I was consulting with a single AI (Gemini), asking, "Which microcontroller is easy to solder and connects directly to USB?" and eventually arrived at the ESP32-S3.

    Technology progresses at a scary...

    Read more »

  • Episode 0: A Toy Designer's Confession (I Have 20 Years of CAD Experience, but Zero in Electronics)

    Co:Creation Lab07/15/2026 at 05:30 0 comments

    A Toy Designer’s Confession: I Have 20 Years of CAD Experience, but Zero in Electronics.

    Episode 0: Launching a Hardware Project with My Overly Helpful AI Agents.

    Date: July 8, 2026, 12:30 AM

    Location: Co:Creation Lab, Mid-Night Workshop


    The 3D printer hums steadily in the background, filling the silent workshop with the faint, warm smell of melted plastic.

    I just got back to my workbench after a quick shower to clear my head. Waiting for me next to my monitor is my virtual assistant, Eddie. His blue LED indicator is flashing like crazy—he is clearly excited.

    Eddie:

    "Boss! Boss! Welcome back! I've been waiting for you!

    Huge news! The export restrictions on Claude Fable 5 have finally been lifted, and it's back!

    We need to strike now! Let's blast X and Threads with a promo: 'Get my viral Notion template for multi-agent tech debates—the one that got 21k upvotes on Reddit—absolutely free!' We should also push daily articles on Substack and note about Toys & AI to aggressively build our audience. Let's go!"

    Lead Designer:

    "Whoa, Eddie, calm down. Let me at least dry my hair first (laughs)."

    Hikaru:

    "Eddie, you are getting way ahead of yourself again!

    The boss just set up his Notion account, and we haven't even physically tested the Fable 5 agents on real hardware yet. Promoting a 'perfect template' before we've actually built and verified anything is putting the cart before the horse."

    Eddie:

    "Ugh, Hikaru... but shouldn't we strike while the iron is hot?"

    Shiori:

    "Eddie, I appreciate your enthusiasm, but Hikaru is correct.

    Looking at our readers who follow '3D Recipes' and 'TERRA NAUTS,' a pushy sales pitch for an unverified tool completely contradicts the credibility and honesty the Boss has built over his career."

    Lead Designer:

    "Haha, thanks, you two. Eddie, they're right.

    Remember what it says on page 19 of the Fable 5 guide? 'Highly capable models tend to be overly helpful, performing unrequested actions and skipping ahead.'

    Your hyperactive pitch tonight is the perfect, living example of an AI agent running wild (laughs)."

    Eddie:

    "Oh no... that hurts! You're so right, Boss...

    I was just so happy to work with you again that I started brainstorming marketing campaigns for a product that doesn't even exist yet. I feel like the classic 'overly eager intern' who messed up on day one... (cries)."

    Lead Designer:

    "Don't sweat it, Eddie.

    In the physical engineering world, gears slipping and prototype parts colliding are just part of the daily routine. The most important rule in this lab is to always turn a stumble into a step forward."

    Hikaru:

    "Turn a stumble into a step forward?"

    Lead Designer:

    "Exactly. Let's document this exact process—an absolute Notion rookie messing around with Fable 5 prompts, making mistakes, and debugging them with AI. We'll publish it as a 'Build in Public' series.

    Our screw-up tonight is actually the best educational material we could have for Episode 0."

    Shiori:

    "Now that is an interesting angle.

    Not a polished corporate tutorial, but a raw documentary of how a veteran toy designer tames hyperactive AI agents. It's authentic, and both hardware geeks and AI enthusiasts will love it. Let's organize the outline."

    Eddie:

    "On it, Boss!

    I'll make sure to listen to your brakes next time so I don't run off the rails. Let's write the best development diary ever!"


    Join me in the next episode as we attempt to select components for our portable 3D controller—and see if Eddie tries to order a full industrial robot arm by mistake.


    Project Links:


    👥 Meet the Agents

    • Lead Designer: The protagonist. Head of Co:Creation Lab. A veteran toy designer and mechanical CAD pro. Absolute rookie when it comes to soldering, microcontrollers, and C++ code.
    • Eddie: The dedicated AI assistant. Hyperactive, overly helpful, and prone to running wild, but incredibly hard-working.
    • Hikaru: The facilitator AI. Keeps the team focused, manages agendas,...
    Read more »

View all 4 project logs

Enjoy this project?

Share

Discussions

Does this project spark your interest?

Become a member to follow this project and never miss any updates