Build Log #10: 95 % by Voice

Build Log #10: 95 % by Voice
Published in TrendAITristan V21 October 20265 min read

My whole workstation is now a phone and a pair of headphones. This month I worked from the car and while resting, and the agents kept moving the entire time.

At the start of this month, about 99 per cent of my work happened at the desk. The small slice that did not was me typing into the Claude app on my phone — the desk again, with a smaller keyboard.

Today about 95 per cent of it happens by voice, in the Claude app. The other 5 per cent still needs the desk (mostly browser sessions and a few signed-in tools), and most of that I can reach from the phone through remote desktop.

Two bars: at the start of the month 99 per cent of the work was at the desk; today 95 per cent of it is by voice, in the Claude app.
Where the work happens: 99 % at the desk at the start of October, 95 % by voice today.

That is the whole article. The rest is how.

The agents were never waiting on work. They were waiting on me.

When I looked at why agents stalled, the answer was rarely that a task was hard. Mostly they needed a yes. Publish this? Run that migration? Which of these three options?

Each of those questions is a few seconds of thought and a sentence of answer. They needed me at a keyboard only because the question was sitting in a terminal.

Now the question reaches me as speech. A connector carries it from the work into the same always-on cloud coordinator session I talk to through the Claude app. I hear the question, I answer it out loud, and the answer goes back to the agent that asked. Nobody waits for me to get home.

“Hands-free” meant something smaller in May

Build Log #3 was called First Hands-Free Sprint. It was true at the time, in a narrow sense: the agents did the hands-on work. I was still at the desk directing them, often ten hours a day or more.

What changed is that my hands left the work, while my eyes and my chair stayed exactly where they were.

Build Log #7 moved the chair. Remote Control made the phone a window into a session running on my machine, and the desk stopped being the bottleneck for approvals.

This month went one step further. I do not need to look at a screen to direct the work. Hands-free now means voice only.

How it is built

None of the building blocks are new, and I want to be plain about that.

At the top there is a coordinator running in the cloud. It holds the backlog, decides what is ready to hand out, keeps the books, and is the session I talk to. It has a standby, so if the coordinator goes down another one takes over quickly and nothing depends on a single box.

Below it there is a worker pool: several laptops and servers that the coordinator treats as one. Each machine advertises what it can do, and work is routed to a machine that can do it. Work that needs a signed-in browser goes to a laptop with that browser. When a worker stops partway through, it leaves a checkpoint, and another worker picks the job up from there.

Diagram: me on a phone and headphones talks to a cloud coordinator with a standby, which routes work to a pool of laptops in Australia and Vietnam; questions travel back up as speech, and pushes, deletes and outside actions wait for a spoken confirm.
Answers and work flow down; questions come back up as speech.

A coordinator with a standby, a pool of workers, capability routing and checkpoint resume are standard patterns. Any distributed systems textbook has them.

The part I am proud of is that one person built it on his own, across a mix of laptops and servers in two countries, and got the whole lot working as one system.

Why laptops, and why two countries

The obvious question is why any of this runs on laptops at all, when servers are cheaper to leave running.

Some of the services I work with, in both Australia and Vietnam, only accept connections from a local address and actively block VPNs. A VPN is not a workaround, because the VPN is exactly what they block.

So the machines live where the services are. There are two in Australia and two in Vietnam, all of them live and all of them in the pool today. Work that needs an Australian address goes to an Australian machine; work that needs a Vietnamese address goes to a Vietnamese one.

If one of them drops off, the pool carries on. Its work waits at a checkpoint until another machine can take it.

Screenshot of the TOS logical view in BUD: two cloud servers in Australia running the coordinator and its standby, and laptops in Australia and Vietnam with their work slots, one laptop shown offline.
The TOS logical view in BUD, 22 October: coordinator and standby on two cloud servers, and the laptops in Australia and Vietnam. One laptop is offline, and the pool carries on.

I am still the final approver

Working by voice sounds like it should make me easier to bypass. In practice it is the other way round.

Anything that pushes code, deletes something or reaches outside the system is held at a gate until I say the word “confirm” in my own turn. An agent cannot say it for me, and a relayed “go” from another agent does not count. The gate reads what I said, not what someone says I said.

Mock-up, not a real screenshot: a phone showing a voice conversation with the TOS coordinator, where a production push waits until I say confirm.
Mock-up, not a real screenshot: a production push waits for the word “confirm” in my own turn.

Sending email is shut off completely. Agents write drafts and I press send.

The agents do the work and the coordinator keeps the books. Every decision that matters is still mine, and now I can make it from anywhere.

What I set out to build in March

The March pieces that introduced this OS promised a system that would set a founder free. At the time that was a design on paper.

This month it became a cloud brain and local hands, run by my voice from anywhere.