Agents in production rarely receive clean text. They receive a phone call, a photographed invoice, a screenshot of an error, a PDF exported from a word processor a decade ago. Multimodal capability is not a feature here; it is the condition of the input.
Routing beats a universal pipeline
The naive approach applies one extraction path to everything and fails quietly on most of it. The expensive approach sends every page to a vision model and works. The useful approach classifies first - cheaply - and reserves the expensive path for input that needs it.
| Input | Strategy | Why |
|---|---|---|
| Digital text, single column | Text layer extraction | Exact and effectively free |
| Multi-column or tabular | Layout-aware extraction | Reading order and cell structure both matter |
| Scan of printed text | OCR | There is no text layer |
| Photo, skewed or handwritten | Vision model | OCR degrades badly; vision tolerates skew |
| Diagram, chart, screenshot | Vision with a description prompt | The content is not text |
| Speech | Streaming STT | Realtime constraints dominate |
Structure is where the accuracy is
The largest single improvement in our document work was not in extraction. It was keeping table headers attached to their rows during chunking, so a retrieved row still says which column a number came from. A retrieved chunk reading 89.00 is retrievable and useless.
The same principle generalises: preserve the relationships the artefact encodes - reading order, cell membership, section hierarchy, page provenance - rather than flattening everything into a string and hoping the retrieval model recovers them.
Vision as the interface of last resort
Vision models can read anything a person can see, which makes them tempting as a universal adapter. They are also the most expensive option per step and the hardest to ground: an answer derived from a screenshot cannot cite a line number. We treat vision as a fallback with a cost attached, and we count how often it fires - a rising fallback rate is a signal about the inputs, not about the model.
Several modalities at once
The direction that interests us is not one modality at a time but several concurrently: an agent that hears a caller describing a problem while reading the document they just sent, and can act on both. That combination raises questions a single-modality system never has to answer.
- Which modality wins when they disagree - the caller says "the invoice is from March", the invoice says April?
- How do you keep a synchronous voice turn responsive while a document is still being processed?
- What does the agent say when it has partially understood the artefact? "I can see part of this" is honest and unusual.
Evaluation
Multimodal evaluation is harder than it looks because there is often no single correct answer. Extraction can be scored against ground truth; a description of a diagram cannot. Our approach is verification-based: after ingesting a document, ask questions whose answers are known from the document itself and check that retrieval finds them. It catches the silent failures - a scan that OCR'd to noise still produces chunks, and without verification those look exactly like success.
Open questions
- Where is the crossover at which routing complexity costs more than sending everything to a vision model?
- How should an agent express partial understanding of an artefact without either overclaiming or becoming useless?
- Can verification questions be generated from a document automatically, rather than written by hand?