PMaker home
Search
中EN
System
Color theme
空格的键盘
空格的键盘空格的键盘产品经理 · AI Builder
Drift isn't the model turning bad; it's a loop losing its constraints in three placesGoal driftAs the context grows, the model starts treating subtasks as the goalStop it: endpoint conditions, high-attention placement, permission to declineTool misuseToo many vague tools, or external content steering it to the wrong callStop it: clear tool docs, tier risky tools, validate args at executionNon-converging loopFailures don't carry a reason, so it keeps retrying in placeStop it: hard step cap, retry with the failure reason, convergence ruleEvery restraint comes from the constraints you build, not from the model waking up

Sort runaway behavior into three buckets, and "the agent is unreliable" becomes three engineering problems you can fix one at a time.

Three Ways Agents Drift

Goal drift, tool misuse, and loops that don't converge — each with its own way to stop it.

Agent failure is not mysterious; it can be classified. Most runaway behavior falls into one of three patterns, and each has a countermeasure you can build. anthropic-agents

Symptoms you'll recognize:

  • It starts on task A, drifts into B, and keeps going after B is done.
  • It calls a tool it shouldn't, or passes absurd arguments.
  • It retries the same action endlessly while the bill climbs.

Goal drift

The goal is clear at first. As the context grows longer and the steps multiply, the model slowly starts treating "finish the current subtask" as the goal itself. You ask it to look up an order; it looks it up, then analyzes the order data, then writes improvement suggestions. Every step is reasonable, but the whole thing has run away.

Why does this happen? Every round, the context is stacked with intermediate results, and the original goal ("look up the order and report") gets buried behind them. Attention drifts toward the most recent steps, so the initial objective fades from view.

Interceptions:

  • Write the goal as an endpoint condition. Not "look up the order" but "look up the order, report it in one paragraph, then end." Give the model an explicit definition of done.
  • Pin the goal somewhere visible. Restate the task boundary in the system prompt or at the top of each round, relying on high-attention placement rather than on the model "remembering" on its own.
  • Let it decline to extend. Spell out: "If the current action goes beyond the original request, stop and confirm with the user." Make stopping a legitimate option instead of forcing it to push on.

Tool misuse

Symptoms: it calls a tool but picks the wrong one, or passes wrong arguments. The dangerous case is "within reach but misused": a tool that can delete data sits next to a read-only one, and the model reaches for the destructive option.

Why does this happen? Two reasons. Tool catalogs are too cluttered and descriptions too vague for the model to tell them apart. And external content (web pages, documents) can hide instructions that steer it toward a specific tool — the classic prompt injection move.

Interceptions:

  • Document the boundaries in each tool description. "This tool is read-only and makes no changes." "Only call this when the user explicitly asks to send." The model follows the description; the clearer it is, the fewer mistakes.
  • Tier the risky tools. Side-effect tools (send, delete, transfer) go in their own group and require human approval — never auto-called inside a loop.
  • Validate arguments at execution time. Don't trust the model's parameters. Check type, range, and permission before the action actually happens.

Loops that never converge

Symptoms: the same action retried over and over. Fail, change a parameter, fail again, change again. Every round adds context and costs money, and the loop never reaches a terminal condition.

Why does this happen? Often the failure result doesn't explain why it failed, so the model has no information to change strategy and just spins in place.

Interceptions:

  • A hard maximum round count. Non-negotiable. This is where loops run off the rails — after N rounds, terminate and degrade gracefully.
  • Retries must carry the reason. "When you retry, include why the last attempt failed." Give the model information to change approach, not to guess blindly.
  • A failure-convergence rule. If the same step fails K times, stop and explain the problem to the user. Don't let it fail a hundred different ways.
  • Monitor rounds and spend. Loop count, tool call count, cumulative tokens — the most basic runtime metrics. Without them you can't even tell how far it ran.

The universal fallbacks

Each pattern has its own fix, but a few rules apply to every case. openai-agents

  • Permissions are the last gate. No set of rules catches every accident. Put every high-risk action (delete, send, pay, export) behind a human checkpoint. It's the one line of defense that doesn't require trusting the model.
  • Log everything. Input, output, tool call, and reasoning for every step. Once an agent has drifted, without logs you can't even explain why it did what it did.
  • Leave an exit for "I don't know." Make "when unsure, stop and ask" a legitimate option in the prompt. An agent that never stops is far more dangerous than one that asks.

The takeaway: intercepting drift is designed, not wished for. The model will not wake up on its own. Every restraint comes from the constraints you build around it.

References

  1. Building effective agents — Anthropic
  2. A practical guide to building agents — OpenAI