When a system has no API, its user interface is the only interface. Computer use agents operate that interface the way a person does - observe, act, check what changed - and the research question is less about whether they can and more about what has to be true before they should.
Three ways to perceive an interface
| Representation | Strength | Weakness |
|---|---|---|
| Raw DOM | Complete and precise | Overwhelmingly irrelevant markup |
| Accessibility tree | Compact, semantic, names match what a user sees | Only as good as the site's accessibility |
| Screenshot | Works on anything visible, including canvas | Expensive per step, nothing to ground on |
The accessibility tree is the right default with a screenshot fallback. How often that fallback fires is a surprisingly good proxy for how well-built the target application is - and, in business software, it fires more than we would like.
From RPA to goal-driven navigation
Classic RPA records selectors and replays them. It is exact, cheap and brittle: a layout change breaks it and it fails loudly. A model-driven agent works from a goal and re-derives the path each run. It survives layout changes and fails *quietly*, which is a materially different operational problem.
RPA breaks in a way you notice. A browser agent breaks in a way you find out about from a customer.
API versus UI
There are four routes into a business system, and they are strictly ordered: a purpose-built API tool, a protocol integration, UI automation, and a human. Each step down loses a guarantee. An API has a contract; a protocol has a schema; a UI has neither and can change overnight without notice.
Our own testing pushed us to a narrower claim than the demos suggest: read autonomously, propose writes for confirmation. Read-only tasks survived layout changes as hypothesised. Write tasks produced occasional plausible-and-wrong entries with no signal that anything had gone wrong, which in a portal that files something with an authority is not a tuning problem.
Containment
An agent driving a browser can navigate anywhere, submit anything and act as its authenticated user. The engineering that makes that deployable is about limits, not capability.
- An isolated browser profile in a container, per run.
- Credentials injected by the runtime, never placed in the model context - the agent knows a login step exists, not what the password is.
- A domain allowlist that applies to every non-read action.
- Classification by consequence: read, write, irreversible - with a human gate on the third.
- Verification after every step that the intended effect occurred, not merely that the click succeeded.
Open questions
- How much of the observed brittleness comes from the model and how much from missing accessibility semantics in the target application?
- Does a cached, per-site "learned map", re-verified on each run, recover RPA's reliability without its fragility?
- What is the right interface for approving a proposed write quickly enough that the human gate does not destroy the time saving?
We exclude targets with bot detection rather than working around them, and we treat the terms-of-service question as a business decision that engineering does not get to answer.