Judge Trustworthiness Starts With the Input

A bad eval number is usually not the judge model's fault: the input was rebuilt from production logs, with internal tool prompts, mismatched vocabulary, and a polluted denominator.

An automated eval produces a pass rate. Do you believe the number? The reflex is to tune the judge prompt or upgrade to a stronger model. The step that actually distorts the number sits earlier: the conversations you feed the judge were rebuilt from production logs, and production logs were never designed for evaluation.

Taobao's AI shopping assistant runs this at industrial scale, scoring a huge volume of real sessions daily. Their conclusion: the bottleneck on eval trustworthiness is not the judge model, it is data quality and the continuous calibration of the standard. taobao-shopping-eval Offline eval data is constructed carefully, with complete fields and consistent formats. Production logs are not — vague user phrasing, abnormal tool returns, fields scattered across tables. The same judge prompt that works on offline data parses mostly to nothing on production data. Online and Offline Evals covers how the two tracks divide the work; this piece is about rebuilding the online side.

From raw logs to input a judge can consume

Production data passes four cleaning layers before a judge sees it. They were not designed up front; each surfaced as evaluation spread across domains, and each was added in turn.

First, complete the fields. Each skill ships its own card protocol and parses fields its own way, so reusing the offline prompt leaves most rows empty. Every domain needs its parse repaired against the offline data definition and its field mapping filled in. One concrete trap: a card can carry thousands of characters, and the assembled context overflows the window. Some platforms do not error on that — they silently return an empty result, which means that row was never scored. Length pre-checks and truncation belong in data preparation.

Second, align the meaning. A user says the parcel is already picked up; the system stores an English enum value that does not mean "shipped". A judge without that mapping reads the agent's "shipped" as contradicting the system state and marks the reply wrong. The fix is an explicit map between user vocabulary and system vocabulary inside the prompt. That is not a judge problem — the input was never translated.

Third, strip the noise. Tool call records carry the tool interface's own internal prompt text: system instructions and developer notes. The agent did not write that, but a judge reads it as something the agent said and scores accordingly. Deterministic rules remove the pass-through text and keep only the arguments and returns actually used. That single step lifted evaluation accuracy across every domain.

Fourth, narrow the intake. Early filters were broad, and plenty of requests that did not belong to the feature landed in the denominator. In one process-oriented skill most samples were users merely asking about policy without initiating anything, or asking about something else entirely; with a polluted denominator the pass rate is both very low and meaningless. Precise filtering on the skill identity made the data usable — and revealed that the real cause of that low pass rate was a prompt that did not fit production data, not a bad product.

None of the four layers is about the judge model. All four are data engineering.

More context can make it worse

A counterintuitive finding: giving the judge more context is not better.

In multi-turn sessions the first version handed over the full conversation history plus every tool call in the chain, on the theory that more background is safer. The judge then used a previous turn's tool return to score the current reply — an amount from an earlier turn treated as today's ground truth — and turns contaminated each other, dragging the pass rate down. Sending only the current turn's tool calls made the pass rate jump.

The reason is attention. When several turns of tool calls sit in one context, the model struggles to tell which turn it is judging and drifts to its neighbors. The prompt-engineering lesson: feed precisely, do not pile up indiscriminately.

A more basic version of the same problem: the judge has no idea what time it is. "Arriving tomorrow" in an agent reply, "within 3 business days" for a refund, "orders in the past year" in a user request — the judge cannot check any of them. Stitching the user's exact timestamp into the input and declaring it the reference clock fixes the stale-timing misjudgments in time-sensitive skills and the mass of misjudgments in retrieval skills caused by a missing time anchor.

When the number looks wrong, attribute before you act

The first instinct when evaluation misbehaves is to edit the prompt. But the root cause sits at one of four levels, and fixing the wrong level makes things worse.

• Data layer: the input itself is wrong — internal prompt text left in, a long card truncated past the crucial detail, a field mapped to the wrong slot. Fix the cleaning logic; it affects every similar row in that domain.
• Prompt layer: the data is right, but the judge misreads the rule — a boundary case has no worked example, or the line between two sub-items is fuzzy and the judge files A's failure under B. Add the rule or the example.
• Standard layer: the evaluation standard itself disagrees with engineering reality. The product intends "no results found, fall back honestly and point to the detail page" while the standard scores that as a failure. This level has the widest blast radius: the offline and online standard snapshots must both change.
• Genuine product defects: whatever remains after ruling out the three above — the agent really did fabricate, really did pass the wrong argument, really did cross a red line. Only these go back to the product team.

The reported lesson is blunt: before attribution existed, a falling pass rate triggered prompt edits, and sometimes the number got worse, because the cause was in the data or the standard and the prompt edit only hid it.

Make hallucination raise its own hand

Attribution is after the fact, and one class of problems never even gets that far: hallucination throws no stack trace, and every traditional health check stays green — downstream services consume the fabricated recommendation as usual, and only much later does someone notice the fact behind it never existed. One large retailer's inventory platform saw roughly 15% of agent responses be hallucinations at the end of its first month in production; six months later, 1.5%. What moved the number was not a stronger base model but treating LLM capability as platform infrastructure llm-platform-playbook.

Two concrete moves. First, enforce structured output at the platform boundary, not inside each application: every call declares an output schema, results are returned only when validation passes, and on failure the validation error is fed back to the model as context for a retry with a hard cap. At the platform level, callers can depend on the contract "returned means compliant", and schema violations become a platform metric instead of a runtime exception buried in application logs. Second, give the output schema a self-reporting evidence field: the model must state whether every claim is backed by retrieved evidence and whether the question falls outside its knowledge. The field costs nearly nothing to generate, yet turns a confident-sounding answer into a machine-readable failure signal — a claim of evidence with empty sources, or a failed validation, is the highest-precision hallucination signal the platform has.

Retries must be classified too. Schema violation: retry with the error attached. The model admits it lacks evidence: retry with stronger grounding context. Rate limits and timeouts — infrastructure failures: back off exponentially. Collapsing all three into one generic retry loop is wrong; blindly rerunning a hallucinating prompt yields another hallucination.

Decompose the metric before diagnosing it

The same discipline applies to online metrics. One domain's routing pass rate fell sharply, and the instinct was a regression — an urgent production bug. Daily decomposition showed the numerator, traces matched to the skill, was flat for days, while the denominator, the domain's base traces, doubled in two days and then nearly tripled. No traffic surge: the domain's share of intent had jumped.

Splitting by sub-label located it. Genuine business actions were flat all along; the spike sat on one label, nearly an order of magnitude up. Those requests read almost entirely as usage questions — how to debug a feature, how to handle an accessory. An upstream skill had scaled up, and the intent model misfiled this class of questions under the domain's complaint label, so the base filter swept them into the denominator. Remove that label and the five-day drop vanishes entirely; the rate returns to normal.

Without attribution the response would have been hunting an urgent online bug. The actual fix was deleting one label from the eval filter because it does not correspond to a real action of that domain.

A rising pass rate does not mean a better product

The goal across iterations is not to make the pass rate go up; it is to make it trustworthy. An early low rate usually means the evaluation method is still wrong, not that the product is that bad. Once the rates across domains settle, every subsequent movement points at a real quality problem.

That distinction decides how you read the dashboard. Before convergence the metric mostly reflects the maturity of the eval system. After convergence it starts reflecting the product. The two reports look identical and mean opposite things.

Calibration runs on continuous sampling. Each iteration ends with an agreement check between the judge and human labels, sampled purely at random so the pass rate stays unbiased. The sample size follows from the precision you need: a confidence interval of plus or minus 1 percentage point takes roughly four thousand samples, and half a point takes about fifteen thousand. One practical corollary: when the pass rate swings well beyond the width of the interval, that is usually not sampling noise but a substantive change in the standard or the model, and worth investigating.

The cost is real too: daily retention of full logs, per-domain cleaning pipelines, and sustained human labeling. For a one-off internal tool this is not worth it. And whether the pass criteria can be written down at all depends on defining done first, which Define Done Before You Write the First Line already covers. A quick check: take a production eval you already run and ask three questions. Was the judge's input rebuilt from production logs? How many cleaning layers does that rebuilding have? When the pass rate looks wrong, do you edit the prompt first or attribute first? If the first one has no answer, the ceiling on that eval is still invisible.