A computer use agent operates software the way a person does: it observes an interface, decides on an action, performs it, and observes the result. The loop is the same as any agent loop; only the observation and action spaces are unusual.
Three ways to see a page
- The DOM. Complete and precise, and mostly irrelevant - a modern page is overwhelmingly markup the task does not care about.
- The accessibility tree. Compact and semantic: roles, names and states, roughly what a screen reader conveys. Best default, and only as good as the site's accessibility.
- A screenshot. Works on anything visible, including canvas-rendered widgets. Expensive per step and provides no text to ground on.
Most practical systems use the accessibility tree with a screenshot fallback. How often that fallback fires is a decent proxy for how well-built the target application is.
Acting is easy; verifying is not
Clicking and typing are trivial. The difficulty is confirming that the action had the intended effect rather than merely succeeding - a click on the wrong element succeeds exactly as loudly as a click on the right one. Every step should assert its expected consequence before the next one is planned.
Containment is the real work
An agent driving a browser can navigate anywhere, submit anything and authenticate as its user. The engineering that makes this deployable is not about capability, it is about limits: an isolated profile, credentials injected by the runtime and never placed in the model context, a domain allowlist for anything that writes, and human approval for anything irreversible.
If an API exists, use it. If it does not, the agent may read autonomously and should propose writes for a human to confirm. Our own browser agent experiment is where that conclusion came from.