Most "AI agent on a phone" content is a demo reel: one clean voice command, one flawless run, cut before anything breaks. I wanted an actual test log instead, so I ran Aiden through 12 tasks people actually do on a phone and tracked what worked, what needed a human in the loop, and what the HID-plus-HDMI approach specifically buys you versus an app-based assistant.

Quick reminder of the mechanism, since it explains most of the results below: Aiden captures the target's screen over HDMI (TC358743 bridge) and drives it over USB HID, a composite gadget exposing keyboard and pointer. No app on the phone, no API, no accessibility permissions. It sees the screen and sends input the way a person would.

The 12 tasks, and what actually happened

Reminders and task capture. Trivial for any voice-driven agent, Aiden included. Parse the utterance, write the reminder. This one barely tests the hardware at all, it's almost pure model capability.

Calendar scheduling. Works, but exposed the first real gap: without an accessibility tree, there's no clean signal for "did the invite actually send." Aiden re-reads the screen after the action to confirm, which is the whole reason perception and action are on separate channels here. An input-only approach (HID with no capture) would be flying blind on this one.

Email triage and reply drafting. Fine for drafting. I kept a hard confirmation gate before send, on purpose, this is exactly the kind of task where "the demo worked" and "the agent is trustworthy" are different claims.

Message summarization. Worked well on apps with a scrollable, readable thread. Struggled on anything using ephemeral or animated message rendering, since a single frame capture can miss content that's mid-animation.

Research summarization. No device control needed here really, model quality more than mechanism, so unremarkable, in a good way.

Meeting notes. Solid on audio capture and summarization. Speaker attribution is still rough, consistent with what most transcription pipelines report.

Travel planning. Multi-app, multi-step, and this is where state going stale mid-run showed up: prices and seat availability change between the agent reading the screen and acting on it. Re-verification after each step matters more here than anywhere else on the list.

Form filling and app navigation. The interesting one. This is the task that actually separates execution models. A huge number of real apps expose no API and no usable accessibility tree, an app-based assistant simply cannot act there. Because Aiden drives the UI directly over HID, it doesn't care whether the app "supports" anything. It fills what a human would fill. The failure mode that showed up: CAPTCHAs and multi-factor prompts, which no automation approach should try to defeat, and Aiden correctly stops and hands control back.

Shopping and price comparison. Comparison is safe and worked fine. Checkout stayed gated behind explicit confirmation the whole time, no exceptions.

Receipt and expense capture. OCR-dependent, so photo quality mattered more than anything about the agent itself. Garbage in, garbage out, as expected.

Smart-home routines. Straightforward when the target app has a stable "scene" or routine feature. Nothing here touched locks or security systems, that's a hard line regardless of how well the rest performs.

Health and habit tracking. Kept deliberately shallow on purpose, reminders and logging only, no attempt at anything resembling clinical advice.

The pattern that mattered most

The single clearest split in this test was: tasks that only need drafting, summarizing, or comparing worked well regardless of execution model. Tasks that needed to act inside an app with no exposed API were exactly where the HDMI-capture-plus-HID-input approach earned its keep, because it doesn't ask the app for permission.

The two problems that showed up across almost every task, regardless of how well it went:

• Action verification. Confirming success by re-reading the screen breaks the moment a loading state lies, a spinner that resolves to an error, a UI that looks done before the network confirms it. This is still unsolved in general, not just for us.
• Staleness. Anything multi-step (travel, shopping) needs re-verification after each action, not just at the end, because the world keeps moving while the agent is mid-task.

Nothing here is a finished product claim, this is a dev board and an early runtime. But logging actual task-by-task results, including the boring failures, seemed more useful than another highlight reel.

Repo: github.com/AidenAI-IO/aiden-hardware-demo