Stop Making the Model Carry Data

Let the LLM interpret intent and let the system move bytes: externalize large results, keep one representation per fact, and bind parameters across steps.

Let the LLM do the work of interpreting intent. Let the system do the work of moving exact values. Once that division is written into the architecture, a whole class of strange production failures disappears, because they share one cause: you asked a probabilistic system to carry something where a single byte cannot be wrong.

The data-hauling fallacy

A model carrying a UUID will truncate it, confuse it with a neighbor, or invent one. This is not a tuning problem; it is a category mismatch. Recognizing that the user wants to modify a specific order is intent understanding, which models do well. Moving a3f2-...-9c1d unchanged from step three to step nine is exact transport, and the model has no business being on that path.

Arrays make it obvious. Handed a large array to pass along, an LLM performs something like unconscious summarization. It does not forward thirty elements; it picks three to five representative ones, considers the job done, and reports nothing unusual. Twenty-five items vanish and no log line fires. qwen-harness

A hard threshold for externalizing

So set a threshold rather than asking the model nicely: tool output above roughly 8000 characters or ten elements must be stored externally, with only a reference and a bounded summary left in context. Mandatory externalization removes the failure mode at the root. If the data never entered the context, it cannot be unconsciously summarized.

That gives data an explicit lifecycle: tool result, classification, inline or artifact reference, bounded summary, index, on-demand inspection, expiry or archive. When you externalize, keep the source, hash, schema, permissions, and expiry policy alongside it, and tag summaries and truncations, because downstream consumers need to know whether they hold the whole thing.

Pair it with partial-inspection interfaces: outline, search, context, head, tail. With those, the model can glance at the structure and choose a segment instead of either reading everything or guessing.

One fact, one representation

The single-representation rule: a given source keeps exactly one form within a single model call. Never a full result alongside its summary, never a summary alongside a preview, never a full result plus the assistant restating it, never several differently truncated copies.

The reason is practical. Multiple representations make the model spend tokens cross-checking which version is true, and wording differences between two copies can generate outright hallucinations. Along multi-step chains this is a main source of degradation.

Do the check at prompt assembly, not after the run. If the same refId enters a prompt in more than one form, refuse to assemble and raise an alert. Treat it as a compile-time error. Better to fail that step than to let the model produce confident output from dirty input.

Previews must be substrings of the source

One detail lesson deserves its own section. When an LLM rewrites a preview, it changes field names and prefixes, which breaks prefix-match recovery downstream and lets dirty data travel all the way to the end of the pipeline.

So preview fields must be generated as substrings of the original text, never rewritten by a model. The principle generalizes: in a data pipeline, determinism outranks intelligence. Extra intelligence is actively harmful here. You think you are letting the model tidy up the display; what you actually did was let it change the identity of your data somewhere no one was watching.

Parameter binding and collect-once upstream

The right way to move data across steps is parameter binding: the runtime injects the value directly from context, skipping the model entirely. The model expresses intent ("this step needs that order ID from the previous step"); the system fetches the value and inserts it.

Push the same principle up one level and you get collect-once upstream, reuse downstream. A given source of truth is fetched once on the chain: source connector, raw artifact, structured projection, verified summary, downstream reference. Persist the raw payload as a controlled artifact and project the fields downstream actually needs. Do not let several agents independently fetch the same external data, and do not pour a full external payload into a long-lived coordinator agent, where it will sit for dozens of steps, consuming budget and diluting everything else.

Facts are not the model's to supply

Push the same division to the execution side and it stops being about transport: the model should not be supplying policy facts either. The cancellation tool in the τ-bench airline scenario is worth copying line by line. Cabin class, insurance, booking time, segments flown, flight status — every one comes from a server-side query, and the time comes from the server clock (now = server_clock.now()). Not a single fact in the decision path is self-reported ai-agent-book.

The model's own expected_cabin_class and expected_has_insurance arguments stay, with a different job. Filling them forces the model to look up the order and check the cancellation policy clause by clause, which makes them a mandatory checklist. Confronted with an economy ticket and no insurance, it tends to notice the violation while still preparing arguments and never makes the call. The server treats those values as a statement of belief, never as fact, and logs a mismatch whenever belief and database disagree — which doubles as a detector for prompt injection.

Three layers, three jobs: natural-language rules for understanding, tool arguments to force the check, server-side verification against real data so a mistake never becomes an irreversible loss. Only the last one is the gate. The corollary people miss is that independence starts with a different data source, not a different model. Let the model fill in cabin_class and have a second model audit whether it filled it correctly, and the gatekeeper is still decorative.

The cost and how to check

You have to build the plumbing: external storage, expiry and archive policies, indexes tied to source-file versions, and fallback to source search when an index is missing, stale, or low-confidence. Single-step and short-chain tasks do not justify it.

Four checks are enough. Dump a long run and look for any ID or array the model copied out of context. Look for any refId appearing in two forms at once. Compare preview fields byte for byte against the source. Then take each policy fact — cabin, insurance, timestamp — and ask whether a query produced it or the model typed it. Fail any of those and your data is moving on luck.