Streaming gets adopted for the obvious reason - the user sees output sooner - and its real costs surface later, in the parts of the system that assumed they could inspect a complete answer before anyone saw it.
What you give up
- Post-hoc validation. A guardrail that inspects the full answer cannot run before the first token has already been read aloud.
- Clean retries. Half an answer has been delivered. Retrying produces a duplicate; not retrying leaves a truncation.
- Simple error handling. An error at token 300 arrives after the user is already reading. There is no status code for "actually, disregard that".
- Exact accounting. Cancelled generations still cost tokens, and reconciling billing with what the user received requires deliberate work.
When streaming is worth it
| Situation | Stream? | Reason |
|---|---|---|
| Voice conversation | Always | Silence is a failure state; first audio is everything |
| Chat interface, long answers | Yes | Perceived latency dominates; user can read as it arrives |
| Agent tool-calling loop | Partially | Stream the prose, buffer the tool call until complete |
| Structured output for a system | No | The consumer needs a valid whole; partial JSON helps nobody |
| Anything with a hard content gate | No | You cannot un-say a streamed token |
The fourth row is where most streaming regret originates. Streaming a JSON payload to a machine consumer adds parsing complexity and buys nothing - the consumer cannot act on half an object.
Streaming tool calls
A model that streams may emit a tool call incrementally. The arguments are not valid until complete, so there is nothing to act on early. What you *can* do early is start speaking - narrate the tool before its arguments have finished arriving, since the tool name is known first.
for await (const ev of model.stream()) {
if (ev.type === 'text') {
// Prose goes out immediately.
await sink.push(ev.delta)
} else if (ev.type === 'tool_start') {
// Name is known; arguments are not. Enough to narrate.
void speak(narrationFor(ev.name))
} else if (ev.type === 'tool_end') {
// Only now is there something valid to execute.
await execute(ev.name, JSON.parse(ev.arguments))
}
}Cancellation is the hard part
Every streaming system needs a cancellation path, and cancellation is only correct if it propagates all the way: stop the model, drop the output queue, truncate the recorded state to what was actually delivered, and release any tool work started speculatively.
Missing the third of those is the classic bug. The recorded conversation contains an answer the user never received, and every subsequent turn reasons from a history that did not happen.
A defensible default
Stream what a human consumes; buffer what a machine consumes. It is a crude rule and it has been right in every case we have had to decide.