Put Loops and Batches in Code

The model understands and judges; the program loops, filters, retries, batches, and parallelizes. A homogeneous batch should need one round trip, not N.

The model understands, judges, plans, and generates the orchestration logic. Loops, filtering, retries, caching, pagination, and batch execution belong to the program. This is not a style preference, it is arithmetic: issue LLM round trips one at a time and latency, token spend, and the chance of drift all scale linearly with N. N round trips should collapse into one.

Leave uncertainty to the model, determinism to the program

The classic cases are reading the top N files after a search, or running the same tool call against N records. If the model fires step one, waits, then decides step two, you pay for that context N times and hand it N opportunities to wander.

The criterion is blunt: a homogeneous batch should need about one model round trip, not N. What the model produces is orchestration logic — it selects a template, or generates a snippet — and what actually runs the loop is your program. Search, read, filter, batch, and retry all happen in code, and the results come back to the model once.

When parallelism is actually safe

Five conditions must hold together: inputs are already determined, no step depends on another step's output, no shared mutable writes, outputs land in independent namespaces, and failures can be handled independently. Break any one and you serialize, or you remove the dependency first.

So the first step is a dependency graph. Batch the independent connectors, tools, subruns, and test jobs and run them together. The graph is not an optional optimization: it is the only evidence you have for whether two calls may overlap, and it is the input to the concurrency limits below.

Cap concurrency per resource, not globally

Parallel is not the same as unbounded. Models, external APIs, browsers, databases, and CI each have their own capacity and each needs its own cap. Most production incidents in this area are not parallelism bugs; they are concurrency taking a downstream service down.

One rule is easy to break: do not re-fetch the same context in order to parallelize. If five branches each read the same file, you hand back the time you saved and pay five times the tokens. Fetch shared context once and distribute it as input.

Orchestration has to land in a trace

Orchestration runs in code, but it cannot be a black box. Every run goes into a trace that supports replay and audit: if a batch touches two hundred files, you must be able to answer afterwards who changed file 137 and on what basis.

The Tarot platform at Taotian draws that boundary worth copying. File changes, commands, mocks, screenshots, diffs, and test results go into an execution ledger as reviewable facts; rationale and residual risk travel as comments and handoffs, and a handoff counts as a lead only, never as a fact. The orchestration layer produces evidence, the model produces judgment, and the two must not share a field. taotian-loop

If the model writes the orchestration code, sandbox it

Offer orchestration templates or an SDK first and let the model pick a template and fill parameters. If you let it generate orchestration code freely, execute it in a sandbox with permission constraints: filesystem scope, network egress, an allowlist of callable tools. A loop the model writes can delete what it should not, exactly like a command it writes.

Anthropic learned a related lesson with long-running agents. A while true loop kept Claude working indefinitely until one Claude ran pkill -9 bash, killed itself, and ended the loop. Side effects like that are inherent to long-running systems. Containers and permission boundaries lower the probability, they do not remove it; tests and CI are what actually hold the quality line. anthropic-c-compiler

The scheduler should read nothing

Break one development task into ten phases — clarify, design, decompose, test plan, review, implement, code quality, end-to-end, acceptance, archive — with one subagent per phase, and the most counterintuitive rule that follows is this: the role doing the scheduling reads no artifact bodies at all.

It does three things. It takes the conclusion (passed, sent back, needs a human) from the structured protocol each subagent returns. It takes progress from the state file. And it relays messages in both directions. The spec, the design, the task list, the test report, the business code — none of it enters its context. Ask it why a conclusion was reached and it will not go read the code to answer; it hands the question back to the subagent that produced the conclusion. baidu-harness-workflow

Why so strict? Because the scheduler's context is the one thing in the pipeline that has to survive to the end. A request that walks through ten phases with several rounds of rejection will have diluted a scheduler that absorbed every artifact along the way — that is exactly how context sours. Keeping the scheduler light is what lets it keep judging reliably across the whole lifecycle.

Two side benefits come with the discipline. Subagents that only recognize named input fields and know nothing about a particular project layout mean switching projects requires only swapping the knowledge base, with zero edits to the phase definitions. And knowledge changes happen in one place instead of ten agent files.

The cost is that you have to define a structured bidirectional protocol, and accept an interaction style where the scheduler does not answer detail questions.

Rollback must be bounded, and state writes must be timed

A phase pipeline rolls backward: a user overturns an earlier decision, a downstream phase finds a gap upstream, a gate rejects, a test fails. Four different triggers, but one shared mechanism, or rollback degrades into jumping around. baidu-harness-workflow

Two constraints pin it down.

First, the target must be earlier and must have actually completed. The target phase has to sit before the current one in the flow table, and it must have genuinely completed earlier in this request's history. That blocks leaps like going from implementation straight back to requirement clarification — the phases skipped in between never ran, so returning to one means starting over from nothing.

Second, the transition completes in a single write. The current phase lands in its terminal state, the target phase flips from complete back to in progress, the pointer moves, and the rollback reason is attached to the target's input so it knows what it came back to fix — all in one write.

There is a timing detail that is easy to miss: while waiting for user confirmation, the current phase is still in progress. The terminal state lands only in the write that actually transitions. The reason is cold-start recovery. If the process dies and restarts while the state file already says "complete" but the user never confirmed, recovery will skip a step that was never finished. Holding the invariant that at most one phase is in progress at any moment, and that it always equals the current phase, is what keeps recovery correct.

The program finishing is not the task finishing

Once orchestration moves into code, a new state becomes easy to misread: the run succeeded.

With today's hosted orchestration implementations, a run reporting "completed" only means the program reached its end. Three things can be true inside it: a branch's task failed, a branch was never created, or the program caught a failure and kept going. All three still finish as "completed". anthropic-workflows

The consequence is concrete: a batch review has produced its summary report, but part of the material was never read. The report is complete in form, the status says success, and the content is missing a chunk. Acceptance that only reads the status field treats it as fully done.

Correct acceptance ignores the status field and checks three things:

• Coverage — of the N objects that should have been processed, how many actually were.
• Failure list — what failed, why, and whether the program swallowed it.
• Key results — whether every key conclusion in the final artifact traces back to a successful source.

Two rules go with it: errors in the program itself (a thrown exception, a timeout, hitting a count limit) usually land in a different terminal state and must be handled separately from "the task result was incomplete"; and any step that writes externally must guard against duplicate submission, because a retry can create a new execution thread and run the same step twice.

One more common misreading: running out of budget is not an immediate brake. Reaching the limit typically only stops new requests from being issued; those already in flight still finish, and with parallelism the actual spend can exceed the limit. Design as if it were a hard switch and you will underestimate the worst case.

How proactive should the agent be

Orchestration answers how the program runs. A neighbouring question is whether the agent should find problems on its own and decide to start. Long-running work makes this unavoidable, because the user is mostly absent; but "proactive" is not a switch, and two axes are needed to describe it.

The horizontal axis is who defines the problem. From L1 to L3 the user defines it — the user sets the time rule, the user sets the trigger condition, and the agent only executes at the right moment. At L4 the user gives only a goal and the agent works out what is missing, which requires the gap to be computable (the goal is either quantifiable or has a checkable state). At L5 the agent infers the goal itself, and an L4 goal that is left vague slides into L5 in practice qoder-proactivity.

The vertical axis is how far it goes: inform, advise, or act on your behalf. The axes are independent — an L5 agent can still stop at informing, and an L3 agent can act externally. When discussing whether to hand something over, state both positions.

L3 to L4 is the qualitative break, because the ownership of responsibility changes. One set of honest numbers: a self-built personal agent's daily loop produced 276 todos from early July, 138 marked complete, and 126 of those were bulk-closed by hand in a single session — the loop itself executed and closed, zero qoder-proactivity. In the same system, replying to colleagues in group chats on the user's behalf — external and irreversible — was something the user was willing to hand over, precisely because the user wrote the rules, the user drew the boundary, and the user owns the outcome. The todo list, by contrast, is a problem the agent computed for itself; until there is an answer to who bears the cost when it computes wrong and acts on it, people keep it at "suggest."

Two judgments are therefore more useful than any measure of smartness:

• Authorization follows the subtask, not the goal. Handing over "drive this milestone to launch" does not authorize messaging everyone on the user's behalf, nor changing production config — each subtask carries different risk, so authorization must be granted one at a time.
• Whether to hand it over depends on the cost of failure, split into three questions: can it be undone, who does it affect, can the result be verified. Who defined the problem decides where responsibility lands when it goes wrong; the cost of failure decides how much room for error exists. Only the position where both are low is fit for direct execution.

There is exactly one lever for moving something upward: state the goal and the acceptance rules clearly. A clear goal means the agent does not have to guess, turning L5 into L4; clear acceptance rules make "did it get it right" checkable, which pushes part of L4 back into "execute by rule" and returns both the problem definition and the responsibility to the person. The ceiling on proactivity is the part of the goal that humans and agents can align on explicitly.

The cost and how to check

The cost is building and maintaining the orchestration layer itself: dependency graphs, concurrency control, trace replay, sandboxing. For heterogeneous work that genuinely needs the model to re-judge at every step, forcing orchestration makes it slower.

Three questions audit you quickly: is your batch's round-trip count closer to 1 or to N? Do any parallel branches share mutable writes? When a batch fails, can you replay how it failed?