It’s currently 12:35 pm on 9/23 as I (Peixing) write this!

Now that the full pipeline is working, we’ve shifted our focus from simply “does it work?” to “how do we make it reliable?”

Our current strategy is a fast, edge-case-driven iteration loop, and we’re trying to keep each cycle to roughly 30 minutes:

Train → Deploy → Evaluate → Find failure cases → Collect targeted data → Retrain → Repeat 🔁

Instead of blindly adding more data, we’re testing each newly trained model, figuring out exactly where it fails, and then collecting new examples specifically targeting those failures.

Swipe Is Trickier Than It Looks

One of the first things we noticed is that the swipe gesture contains a surprisingly large range of poses. During a swipe, the pointing hand can appear at many different angles and positions.

Because we’re currently doing image-based detection rather than video/temporal detection, the model doesn’t actually understand the motion of a swipe. It only sees individual frames.

That means our dataset needs to contain the different intermediate poses and angles that can occur throughout the swipe:

Swipe motion → many individual poses → all need to be represented in the dataset

This has become an important consideration in how we’re collecting training data.

Swipe vs. Thumbs-Up Confusion

We also discovered an interesting failure case: the model would sometimes confuse a swipe/pointing gesture with a thumbs-up gesture.

Rather than just adding random examples, we deliberately collected hard examples containing both gestures in the same image—one hand performing the swipe/pointing gesture and the other performing a thumbs-up.

After adding these examples and retraining, the confusion was significantly reduced. 🎉

This was a nice validation of our rapid iteration strategy: find a specific failure mode, collect data that directly exposes that ambiguity, and retrain.

Current Edge Case: Palm / Stop ✋

Our next problem is the open-palm stop gesture.

We’re seeing cases where the palm is either:

• not detected at all, or
• incorrectly classified as swipe, or
• incorrectly classified as thumbs-up.

So we’re back in the loop again. We’re currently collecting more palm examples covering different angles, positions, people, distances, and backgrounds, especially examples that the current model finds difficult.

Then we’ll retrain, test again, and see what breaks next. 😭

At this point, our workflow is basically:

Don’t guess what data the model needs—let the model’s failures tell us what data to collect.

And with roughly 30 minutes per iteration, we’re trying to squeeze in as many targeted improvement cycles as we can before the final demo.