A browser agent for portals that have no API

Problem

A recurring request in customer projects: "this supplier portal has no API, can the agent just use it?" The portals in question are ordinary business web applications - order status, appointment slots, case files at an authority.

Classic RPA solves this with recorded selectors and breaks whenever the page changes. The question was whether a model-driven browser agent degrades more gracefully.

Hypothesis

A browser agent working from the accessibility tree rather than from pixels or fixed selectors will survive layout changes that break scripted RPA, at acceptable cost for read-only tasks.

Architecture

Three representations of a page were available to us, and the choice between them turned out to be the main design decision.

We settled on the accessibility tree as the primary observation, with a screenshot fallback only when the tree is uninformative - a canvas-rendered table, or a widget with no accessible names. That fallback fired more often than we would like, which is itself a finding about the state of business web applications.

Implementation

The agent runs in a container with its own browser profile. Credentials are injected by the runtime at login time and are never placed in the model context - the agent knows a login step exists, not what the password is.

// Read actions run freely. Anything that changes state on the other side
// needs an explicit allow, and anything irreversible needs a human.
type Risk = 'read' | 'write' | 'irreversible'

function classify(a: Action): Risk {
  if (a.kind === 'navigate' || a.kind === 'extract') return 'read'
  if (a.kind === 'submit' && IRREVERSIBLE.test(a.formName)) return 'irreversible'
  return 'write'
}

async function execute(a: Action, ctx: Ctx) {
  const risk = classify(a)
  if (risk === 'read') return run(a)

  // Domain allowlist applies to every non-read action. An agent that can
  // navigate anywhere must not be able to submit anywhere.
  if (!ctx.allowedDomains.has(hostOf(a.url))) throw new Blocked(a)

  if (risk === 'irreversible') {
    const ok = await ctx.humanApproval(describe(a))  // blocking, with a timeout
    if (!ok) throw new Declined(a)
  }
  return run(a)
}

Verification after each step was the other thing that mattered. The agent must check that the action had the effect it intended - not that the click succeeded. A click that succeeds on the wrong element succeeds just as loudly as one on the right element.

Result

Read-only tasks worked well enough to be interesting. Retrieving a status, extracting a table, checking availability - these survived layout changes that would have broken a selector script, which was the hypothesis and it held.

Write tasks did not, and the failures were not the entertaining kind. The agent would complete a multi-step form correctly nine times and on the tenth pick a plausible-but-wrong option in a dropdown it had never seen before, with no signal that anything had gone wrong. In a portal that files something with an authority, a 10% silent error rate is not a tuning problem - it is a category error about where the work belongs.

We stopped, and wrote down the rule instead: if an API exists, use it; if it does not, the agent may read autonomously and propose writes for a human to confirm. That is a smaller claim than "autonomous browser agent", and it is one we can actually stand behind.

The secondary finding was about economics. Screenshot-based steps are expensive enough that a task requiring many visual fallbacks costs more than the manual work it replaces - which quietly kills a lot of otherwise plausible use cases.

Limitations

What this experiment does not establish. Listed because an experiment without limitations is an advertisement.

• Exploratory. This ran against a handful of portals in one language, in a lab environment, with no production traffic behind it.
• We have no reliable success-rate figure to publish. The tasks were too heterogeneous to average, and reporting one number across them would be misleading.
• Portals with aggressive bot detection were excluded rather than worked around. Evading detection is not a problem we are interested in solving.
• Accessibility-tree quality varies enormously between sites, so results transfer poorly. A well-built portal and a canvas-rendered one are effectively different experiments.
• Nothing here addresses the terms of service question, which is a business decision and not an engineering one.

Next steps

• Formalise the propose-and-confirm pattern into a reusable approval UI, since that is the shape that survived.
• Measure how much of the observed brittleness comes from the model versus from missing accessibility semantics.
• Investigate whether a site-specific "learned map" - cached once, verified per run - recovers the reliability of RPA without its fragility.