Reading what the customer actually sent

Problem

An agent is only as useful as the knowledge behind it, and in practice that knowledge arrives as documents. Not as an API, not as markdown - as a PDF someone exported, a scan of a laminated sheet, or a photo taken at an angle.

The naive pipeline extracts text and indexes it. It works for the first document and fails quietly for the rest: a price table becomes an unordered word soup, a two-column layout interleaves its columns, and a scan yields nothing at all.

Hypothesis

Routing documents by type to different extraction strategies - rather than applying one pipeline to everything - produces materially better retrieval at lower cost than sending every page to a vision model.

Architecture

The classifier is the cheap part and does most of the work: it inspects whether a text layer exists, how much of it there is, and whether the page geometry suggests columns or a table. Only pages that fail those checks reach a vision model.

Implementation

Structure is preserved rather than flattened. A price table is indexed as rows with their headers attached, so a retrieved chunk still says which column a number came from. This is unglamorous and it is where most of the accuracy came from.

// A retrieved row must carry its own context. Without the header, "89.00"
// is retrievable and useless - the agent cannot tell price from part number.
function chunkTable(t: Table, source: SourceRef): Chunk[] {
  return t.rows.map((row) => ({
    text: t.headers.map((h, i) => `${h}: ${row[i]}`).join(' | '),
    // Provenance travels with the chunk so an answer can cite the page it
    // came from - and so a stale document can be found and removed.
    source: { ...source, page: t.page, table: t.index },
  }))
}

Every chunk carries provenance to the page. That serves two purposes: the agent can say where an answer came from, and when a customer replaces a price list, the superseded chunks can be found and deleted rather than lingering to contradict the new ones.

Result

Routing beat uniform processing on both axes. Sending everything to a vision model is accurate and expensive; sending everything through text extraction is cheap and wrong for a meaningful fraction of real documents. The classifier makes the expensive path rare.

The largest single improvement was not in extraction at all. It was keeping table headers attached to rows during chunking - a change of a few lines that fixed a category of wrong answers we had been attributing to the retrieval model.

The related production capability is the knowledge import in bitpull.ai, which ingests websites and PDFs including scanned ones. This prototype is where the routing and provenance ideas are being worked out before they go near it.

Limitations

What this experiment does not establish. Listed because an experiment without limitations is an advertisement.

• Prototype status: this runs offline against a document set we assembled, not as a live intake path.
• We report improvements qualitatively. Our evaluation set is small and drawn from a narrow set of industries, so a published accuracy figure would overstate what we know.
• Handwriting remains poor. Vision tolerates it better than OCR, which is a low bar.
• The classifier is heuristic and will misroute unusual layouts. It fails toward the expensive path, which is the right direction but not free.
• Multi-page context - a table continuing across a page break - is not handled and produces chunks that look complete but are not.

Next steps

• Handle cross-page tables, the most common structural failure remaining.
• Build an evaluation set large enough that we can publish numbers instead of impressions.
• Test whether verification questions can be generated from the document automatically rather than written by hand.