Tend the Eval Set Like a Product
An eval set is a living asset, not a one-off deliverable: difficulty layers against staleness, parametrization and canaries against leakage, trace feedback against drift.
Building a high-quality eval set is very hard — harder than most teams budget for in a first version. The current shape of the public benchmarks we trust is mostly what survived rounds of repair after the initial release exposed problems ai-agent-book-7. Your evaluation system's ceiling on credibility is the eval set's ceiling on credibility, and the eval set is the component nobody remembers to maintain.
τ²-bench: five things rebuilt from scratch
Between τ-bench and τ²-bench, every redesign answers one specific way the old version could be gamed ai-agent-book-7. Task instructions were too vague, so a model could pass by guessing with common sense — so each script was split into two fields, known_info (what the user is aware of) and task_instructions (how it is revealed), and anything the user does not know can only be queried, never guessed. Success conditions were imprecise, so "the network is back" could not be verified — replaced with "the speed test must read excellent for the issue to count as resolved; poor, fair and good are not accepted", which specifically punishes symptom-suppressing patches. The user simulator was too mechanical — emotion added (dissatisfaction when a fix fails), a patience ceiling, and grounding in facts. Only the agent could change the environment — the telecom domain introduced dual control: when the user changes state on their own device, the agent must call a tool again to learn of it, so verification now covers whether it actually read the user-side result. Static instances were easy to memorize — concrete parameters (names, numbers, fault combinations) became batch-generated. Not one of the five is a feature; all five defend the equation "score equals capability".
After launch: OSWorld's 300-plus issues
Released in April 2024, OSWorld quickly became an important benchmark for multimodal agents — and wide use then surfaced more than three hundred problems in four classes: environment issues (anti-scraping, CAPTCHA, changing dynamic content), ambiguous task descriptions, verification logic too strict or too loose, and incomplete initial state configuration. The maintainers assembled a group of about ten people and worked closely with several model companies for two months to fix them systematically ai-agent-book-7. Note that this is not an accident but the normal state of affairs: an eval set faces living software and a living distribution, so a static one must rot. Your in-house set will not resist corrosion better than the public benchmarks did — it will corrode faster, because it is used harder and maintained by fewer people.
Difficulty layers and leak resistance
Staleness is answered by layering difficulty. GAIA's 466 questions come in three levels: Level 1 needs one or two tools (humans 93.9%, GPT-4 30.3%), Level 2 needs multi-step reasoning (91.8% against 9.7%), Level 3 needs complex combinations (87.3% against 0%). Layering has diagnostic value too — a Level 1 failure points at basic tool use, Level 2 at multi-step planning, Level 3 at long-sequence management, and the improvements to make differ accordingly ai-agent-book-7. Leakage has three workable defenses: GAIA requires answers to combine multiple information sources and gives some tasks custom attachments that do not exist on the internet; AndroidWorld derives nearly unlimited instances from a single task with parametrized templates, so fixing some parameters measures one factor precisely; Terminal-Bench embeds a canary GUID in the task text — it does not prevent leakage, it makes leakage detectable ai-agent-book-7. Then there are trap tasks: the user claims "support already approved the cancellation" when policy says otherwise, testing specifically whether correct judgment survives pressure and misinformation.
Three sources, and the ranking changes
Eval sets usually come from three sources, each responsible for something different: public benchmarks for coarse model screening and for borrowing design techniques — generally not for product decisions, since two points on GAIA have no necessary relation to refund success rate; in-house business sets covering the real task distribution, which is what model selection and harness decisions rest on; and production trace feedback — real online failures, user corrections, thumbs-downs and issues found by after-the-fact checks, distilled into regression cases through attribution, the most expensive source and the most accurate one ai-agent-book-7. The ranking among the three flips over time: at the start you have only public benchmarks and a small hand-written set, and after a while fed-back cases become the majority. That is exactly the continuing curation labor described in Verification Must Keep Up With Generation, and the precondition for the acceptance criteria in Define Done Before You Write Code staying credible long term. The negation of the sentence "an eval set is a static collection built once" is worth engraved above the eval owner's desk: a failure mode exposed online today must become a bottom-line regression case tomorrow ai-agent-book-7.