Welcome to the EWNA Team 1 (Team Marker My Words) Hackaday project page! We built this project for the Robotics Challenge (sponsored by Quilter) at the Embedded World North America Hackathon, held at the Anaheim Convention Center. Our goal was a robotic arm that uses on-device AI to pick up a marker, trace a hand-drawn line, avoid obstacles in its path and return the marker to where it started, with all the computing on the Arduino UNO Q.
Every part used in this build, including those provided by the hackathon, is listed in the "Components" section, and a full Bill of Materials with estimated prices is available in the "Files" section. The project is designed to be reproducible and beginner-friendly: if you follow the "Instructions" section step by step, you should be able to build one yourself, from assembling the arm and wiring the servos to setting up the Arduino UNO Q and training the detection model. The "Files" section also contains our code and other useful resources.
We've documented our whole journey in the "Logs" section. That includes the hardware hurdles we worked through, how we hand-labelled more than 750 images to train our marker detection model, and the tips we picked up along the way. We hope it helps you learn from both our wins and our roadblocks.
The Problem
Warehouse robots, delivery bots and drones all share one challenge: they have to carry out precise tasks in spaces that are always changing. People walk past, boxes get moved, and new obstacles show up without warning. These robots can't afford to wait on a cloud server for instructions, and a pre-recorded script breaks the moment the world stops matching it. They need to sense what's around them and decide what to do on their own, in real time. Our project is a desk-sized version of that problem, using a small robotic arm, a marker and a sheet of paper.
Why it's different
Everything happens on the robot itself. We split the work between two boards and gave each one the job it's best at. The Arduino UNO Q is the brain: its Linux processor takes on the heavier work of vision and path planning, turning what the camera sees into a sequence of joint positions. The ESP32 controller that comes with the ACEBOTT kit is the muscle: it receives those positions from the UNO Q [over a serial/UART link] and drives the servos smoothly and precisely. Nothing in the run is pre-programmed either. The arm works out every path fresh from what the camera sees at that moment, so it can handle any line and any obstacle it's given.
Bill of Materials
Prices are in USD and don't include shipping, customs or import fees, which vary by location.
The hackathon organisers provided all of the components except the USB-C to USB-C cable, which we brought ourselves.
The ACEBOTT kit is priced as a single item. The individual prices for its parts are rough estimates for reference and aren't added to the total.
A stable base from our name tags. We cut up the "Hello, my name is" name tags from the hackathon badges and layered them into a small, flat platform under the marker. The sticky backing holds the marker's base in place, and the wider footprint keeps it standing upright. Now it stays put until the arm picks it up, and it gives the marker a clear, consistent starting spot to return to.
Better grip from the kit's leftover anti-slip sleeve. The ACEBOTT kit came with a spare non-slip sleeve that we used during assembly. We slid it onto the marker where the gripper grabs it. The rubbery surface gives the claw much more to hold onto than smooth plastic, so the marker doesn't slide or twist when it's lifted, and it stays pointing straight down while tracing the line.
Bonus: a cardboard camera rig. Our overhead camera mount is made entirely from leftover shipping boxes: a cardboard shelf held up by a box, with craft sticks and electrical tape keeping the camera and its cable in place. It isn't glamorous, but it holds the camera steady at a fixed height, and that's all the calibration needs.
Our system has two jobs: working out where the arm should go, and making the servos actually get it there. The ESP32 handles the second. Here's how the two sides talk to each other.
The planning code works in real-world millimetres on the paper. For each point on the path, it uses inverse kinematics(based on our measured arm lengths) to work out the angle for each joint, adjusts for each servo's individual calibration, and sends the angles to the ESP32 over a USB serial connection as a short text message.
We designed a simple set of commands:
Move (B/S/E/G): set the base, shoulder, elbow and gripper angles. The move's duration is calculated automatically from whichever joint has the farthest to go.
Timed move (T): the same, but with a set duration, for steady, evenly paced tracing.
Home (H) and Stop (X): return to a safe position, or freeze in place.
The ESP32 moves each servo smoothly to its new angle, updating about 50 times a second, and only replies "done" once the move is actually finished. That means the planning code never has to guess how long a move takes: it just waits for the confirmation and sends the next one. If anything goes wrong, the arm goes to a safe "park" position (claw open, pen lifted, home) instead of freezing mid-move.
We also started with a different approach, fitting servo angles by moving the arm onto all 12 calibration dots by hand. Once we'd measured the arm's real lengths, we switched to direct inverse kinematics, which we verified against 500 random target points with zero error.
We realised our first design was ambitious. While tracing, the arm would check the camera every ~20 mm, and if something new had blocked the path, it would re-plan just the remaining route and keep going. We built it and it worked in testing, but the more we thought about the actual challenge, the more we questioned whether we needed it.
The organisers clarified that during the demo, the line and obstacle are placed before the run starts and stay put. With a still scene, watching the camera mid-trace adds extra steps, extra processing and extra ways for something to go wrong, with no real benefit.
So we simplified. Now the arm takes one look at the scene and plans everything up front:
Finds the marker, the line and the obstacle in a single camera frame. The arm is recognised too, so it's never mistaken for an obstacle.
Plans the line, including any detour around the obstacle.
Stitches everything into one complete route: marker → start of the line → along the line → end of the line → back to the marker.
Saves the full plan to a file, with every point labelled as drawing, detour, lift-over or travel, plus an overlay image so we can check it before the arm moves.
Then the arm simply executes the plan from start to finish. Sometimes the best engineering decision is knowing what to take out!
The full planned route overlaid on the camera view before the arm moves.
When the arm meets an obstacle on the line, it has three options, and it tries them in order:
Go around (edge walk). The arm follows the edge of a safety zone drawn around the obstacle until it can rejoin the line on the other side. It's fast (about 1 ms to plan in our tests) and works well for a single object.
Search for a route (A*). If walking around the edge would take the pen off the paper, a grid search finds another way around. It's slower (about 2.8 seconds on the same test), so it's only used as a backup.
Lift over it. If there's no way around at all, the arm lifts the pen above the obstacle and puts it back down on the other side, leaving a small gap in the line. The arm never drags the pen through an obstacle.
Getting this right took a few rounds of bug hunting. We found that the A* search failed completely whenever the starting point sat inside an obstacle's safety zone. We also found that smoothing the final path could nudge it a fraction of a millimetre into the safety zone near sharp corners (up to about 0.45 mm). We fixed both, and added a final check that pushes any stray point back out. Our tests went from 7 safety-zone violations to 0, and we added a special case for when the obstacle covers the entire rest of the path, where the arm simply goes over.
The path planner's colour legend: green = on the line, magenta = detour around, blue = lifted over, red outline = safety clearance zone.
Not every problem is about algorithms. Sometimes the camera just won't cooperate. While testing on a laptop, our C270 webcam started causing trouble in ways we didn't expect.
First, it kept changing its device number. The same camera would show up as camera 1, then silently move to camera 0, with no error and no warning. The wrong camera would open, and every position the system calculated would be wrong. We tried identifying the camera by name from the system's device list, but that list changed order within seconds. We tried its "unique ID" too, but that turned out to change depending on which USB port it was plugged into.
In the end, we went with the most reliable test of all: look at what each camera actually sees. We saved a snapshot from every camera index and checked them by eye. One showed our desk setup, one was the laptop's selfie camera, and one showed only black.
That black frame was our next mystery. The camera returned a completely black image every time, byte-for-byte identical. We ruled out other apps holding the camera, then wrote a diagnostic that tries different video formats and resolutions and prints each frame's brightness values, so we could tell a dead feed from a dark room.
We also learned that the paper needs to be kept away from anything else bright in the camera's view. Once, a monitor in the background merged with the paper, and the software couldn't find the page's edges.
Lesson learned: when hardware misbehaves, check what it's actually doing, and don't trust assumptions about what it should be doing.
Once we had our labelled dataset, we faced a question: should the model learn from colour images or greyscale ones? Greyscale models can be more robust to lighting changes, while colour gives the model more information to work with. Instead of guessing, we trained both.
Our first model, trained on 410 colour images, scored a mAP50 of 0.982. The greyscale version came in at 0.964, which was close enough that neither was a clear winner. We even built a version that automatically switched between the two models depending on the camera feed. After testing both live on the webcam, though, we decided that running two models added complexity without a real benefit, so we went with colour.
Then we pushed further. The judges might pick a different Crayola marker than the one we practised with, so we added 124 new images of markers in different colours, for 534 in total. The score dipped slightly to 0.962, but that was expected: the model was now learning what any marker looks like, not just one specific colour. On real footage it spotted the marker in 18 of 21 frames, and the misses were mostly frames where the marker hadn't come into view yet.
Finally, we trained a combined model that recognises both the marker and the robot arm itself (765 images), which reached a precision of 0.989 and a mAP50 of 0.985. Knowing where the arm is means the system never mistakes its own gripper for an obstacle.
Detecting the marker is only half the job: the arm also needs to know exactly where it is on the table. Our script
marker_position.py
handles this step. It loads our trained detection model, watches the live camera feed and converts each detection into a real-world position in millimetres that the arm can reach for. For every frame, it reports the marker's coordinates, the spread (how much the position varies across recent readings, so lower means steadier) and the model's confidence that it's looking at a marker.
In this test, the readings settle quickly. As the detection locks on, the confidence rises and the spread drops from 2.4 mm to just 0.3 mm, so the marker's position holds steady at about (141.5, 87.4) mm. The occasional "no marker" lines are frames where the model wasn't sure enough to report a detection, and the script simply skips them. Once the reading is stable, pressing s saves it as the pickup position. That's the "remembered start location" the arm uses to put the marker back at the end of each run.
Live output from marker_position.py: as the model locks onto the marker, the confidence climbs from 0.71 to 0.94 and the position settles to within 0.3 mm.
Follow the instructions in the official video from the kit to assemble the arm
2
Understanding the Robot Arm and its Servos
The ACEBOTT QD022 arm moves with four MG90 servo motors, one at each joint. A servo is a small motor that turns to a precise angle (0°-180°) when it receives a control signal, which is what lets the arm move to exact positions. Together, the four joints let the arm reach, lower, trace and grip.
How to connect and test the servos
Plug each servo cable into its matching pin header on the ESP32 board, following the table above. The brown/black wire goes to GND, red to power, and orange/yellow to the signal pin.
Install the ESP32Servo library in the Arduino IDE (Library Manager → search "ESP32Servo").
3
Wiring and Pin Connections
It is important to note which pins to connect the wires coming out of the servos to.
Below is the ESP32 Max V1.0, the controller board that comes with the ACEBOTT kit and drives our arm's servos. The rows of pin headers along the top edge are where the servos plug in. Each column has a signal pin (blue), a 5V pin (red) and a GND pin (black), with its GPIO number printed above. The board can be powered through its USB-C port (5V) or the DC jack (7–15V), and the serial pins on the right (TX/RX) are how it can talk to other boards like the Arduino UNO Q.
Each servo has three wires: brown goes to GND, red to 5V, and orange to the signal pin. This example shows the chassis (base) servo plugged into GPIO5. Connect the other servos the same way: shoulder to GPIO16, elbow to GPIO17 and claw to GPIO18. Make sure the brown wire always faces the black GND row, because a servo plugged in backwards won't move.
This diagram shows all four MG90S servos connected to the ESP32 Max board at once. Each servo plugs into its own three-pin header in the first four columns of the top row (H1–H4), which match GPIO5, GPIO16, GPIO17 and GPIO18: the chassis, shoulder, elbow and claw. Every header supplies power (5V) and ground (GND) alongside the signal pin, so each servo needs just one plug and no extra wiring. Once all four are connected, the ESP32 can control every joint of the arm on its own, taking movement commands from the Arduino UNO Q.
Shubhan Mital
