Online and Offline Evals

An online eval is an alarm, not an acceptance criterion. The offline eval is the evidence that a change actually improved things, and a flywheel connects the two.

An online eval is an alarm, not an acceptance criterion. Treating production trace scores as evidence that a change improved the system is the most common mistake an agent team makes: online signals tell you the number dropped, not whether your change made things better. Only an offline eval answers that. You need both, and a flywheel to connect them.

Two evals, two different questions

Build only one and the failure is predictable. Online only: you see the drop but cannot verify the fix under controlled conditions. Offline only: the dataset drifts from production, and the suite stays green while the product degrades.

The flywheel

Production traces, negative user feedback, and responses that reviewers rewrote in the annotation queue feed the next test set. The offline eval validates the change, you deploy, new traces come back. Rewritten responses are the highest-value rows for the golden set.

One hard constraint: never run evals on synthetic data alone. Synthetic scenarios rarely match the mess of real ones, and an agent tested only on imagined problems handles only imagined problems. For the flywheel to turn, real failures have to reach the test set first.

The judge triages; it does not decide

Human attention is the bottleneck: no team can sustain reviewing large volumes of output by hand, yet an uncalibrated LLM judge cannot be trusted. The workable split is triage: clear, low-risk cases run automatically; low-confidence, contested, and high-impact cases route to a human. Sample high-confidence cases on a schedule to catch systematic errors, since handing everything to a model and watching only the pass rate discards calibration. Track the judge-human disagreement rate, and version and evaluate the judge prompt itself. Humans design and calibrate the automated evaluator rather than reviewing output by hand. langchain-human-judgment

Online also needs keep rate and semantic reaction

Latency, token efficiency, tool call counts, and cache hit rates only show trends; they cannot answer whether the agent did the job well. Cursor added two online measures: cursor-agent-harness

• Keep rate: the share of proposed code changes still in the repository after a fixed period. A low keep rate means users edited the result or kept asking the agent to fix it — low initial quality.
• Semantic reaction: an LLM reads the user's response to the agent's first output and judges satisfaction. Moving to the next feature request is a strong completion signal; pasting a stack trace is a strong failure signal.

Their value shows in a decision: Cursor shelved using a more expensive model for context summarization, because these measures showed almost no quality gain for the cost. Public benchmarks only approximate real usage, so relying on them alone misses signals like these.

Discipline for self-evolving rules

Once the flywheel starts editing rule files such as SKILL.md, one precondition applies: the task output must be objectively right or wrong.

• Only the intersection of a wrong result and an abnormal process triggers a change, and only when one root cause covers a sufficient share of failures, around 30%. A correct result from a messy process stays untouched.
• Four gates: Target, at least one target case improves; Guardrail, no regression among passing cases, weighted several times heavier than no improvement; Holdout, F1 degradation past the threshold rejects the change; Verify, the text quality of the rule file.
• A three-way data split: Selection participates in diagnosis, Holdout monitors generalization, Golden is human-certified and never evolves. If the editor can see the answers, it is memorizing them.
• A taboo list: rejected change signatures, root cause plus diff hash, shared across versions and branches, surviving rollback and injected as negative constraints. Cap the diff size so regressions can be located.
• Measure gains at an equal token budget. Best-of-n and retry-until-it-works gains are just spending more money, not evolution.

Alibaba's SKILL.md evolution work follows this discipline. Diagnosis is entirely deterministic rules, patterns such as claiming a step was executed with no evidence in the session, or repeating a call with identical arguments three or more times. The LLM only transcribes the diagnosis into a diff, because an LLM judging skill quality directly measured near random. Across 63 cases, pass rates moved from 77.8% to 84.1% for Kimi, 77.8% to 88.9% for GLM, and 82.5% to 87.3% for DeepSeek. The same project recorded one more number: swapping the judgment word vulnerability for risk moved accuracy from 89.3% to 62.1%. alibaba-skill-evol