Put Loops and Batches in Code

The model understands and judges; the program loops, filters, retries, batches, and parallelizes. A homogeneous batch should need one round trip, not N.

The model understands, judges, plans, and generates the orchestration logic. Loops, filtering, retries, caching, pagination, and batch execution belong to the program. This is not a style preference, it is arithmetic: issue LLM round trips one at a time and latency, token spend, and the chance of drift all scale linearly with N. N round trips should collapse into one.

Leave uncertainty to the model, determinism to the program

The classic cases are reading the top N files after a search, or running the same tool call against N records. If the model fires step one, waits, then decides step two, you pay for that context N times and hand it N opportunities to wander.

The criterion is blunt: a homogeneous batch should need about one model round trip, not N. What the model produces is orchestration logic — it selects a template, or generates a snippet — and what actually runs the loop is your program. Search, read, filter, batch, and retry all happen in code, and the results come back to the model once.

When parallelism is actually safe

Five conditions must hold together: inputs are already determined, no step depends on another step's output, no shared mutable writes, outputs land in independent namespaces, and failures can be handled independently. Break any one and you serialize, or you remove the dependency first.

So the first step is a dependency graph. Batch the independent connectors, tools, subruns, and test jobs and run them together. The graph is not an optional optimization: it is the only evidence you have for whether two calls may overlap, and it is the input to the concurrency limits below.

Cap concurrency per resource, not globally

Parallel is not the same as unbounded. Models, external APIs, browsers, databases, and CI each have their own capacity and each needs its own cap. Most production incidents in this area are not parallelism bugs; they are concurrency taking a downstream service down.

One rule is easy to break: do not re-fetch the same context in order to parallelize. If five branches each read the same file, you hand back the time you saved and pay five times the tokens. Fetch shared context once and distribute it as input.

Orchestration has to land in a trace

Orchestration runs in code, but it cannot be a black box. Every run goes into a trace that supports replay and audit: if a batch touches two hundred files, you must be able to answer afterwards who changed file 137 and on what basis.

The Tarot platform at Taotian draws that boundary worth copying. File changes, commands, mocks, screenshots, diffs, and test results go into an execution ledger as reviewable facts; rationale and residual risk travel as comments and handoffs, and a handoff counts as a lead only, never as a fact. The orchestration layer produces evidence, the model produces judgment, and the two must not share a field. taotian-loop

If the model writes the orchestration code, sandbox it

Offer orchestration templates or an SDK first and let the model pick a template and fill parameters. If you let it generate orchestration code freely, execute it in a sandbox with permission constraints: filesystem scope, network egress, an allowlist of callable tools. A loop the model writes can delete what it should not, exactly like a command it writes.

Anthropic learned a related lesson with long-running agents. A while true loop kept Claude working indefinitely until one Claude ran pkill -9 bash, killed itself, and ended the loop. Side effects like that are inherent to long-running systems. Containers and permission boundaries lower the probability, they do not remove it; tests and CI are what actually hold the quality line. anthropic-c-compiler

The cost and how to check

The cost is building and maintaining the orchestration layer itself: dependency graphs, concurrency control, trace replay, sandboxing. For heterogeneous work that genuinely needs the model to re-judge at every step, forcing orchestration makes it slower.

Three questions audit you quickly: is your batch's round-trip count closer to 1 or to N? Do any parallel branches share mutable writes? When a batch fails, can you replay how it failed?