Saving Experience Is Not Learning From It
Dumping a hundred traces into a vector store does not perform the cross-case comparison. Learning happens after evaluation, contrast, induction and validation, not at write time.
A deployed model does not change its own weights because it ran one inference. Context engineering lets an agent adapt inside the current task, but when the task ends the adaptation ends with it; and storing conversations in memory is equally not learning new behavior — raw traces are long and noisy, and inside them sit valid strategies, accidental successes, wrong attributions and untrusted inputs ai-agent-book-9. Put a hundred traces into a long context or a vector store and the model can retrieve a case when it needs one, but it will not automatically perform the cross-case comparison: which steps recur across the successful traces, whether this particular success came from strategy or from an environmental accident. Learning happens after "evaluate, contrast, induce, validate" — not at the moment the log is written to disk.
Three layers of data, each with its own job
Raw traces are the wrong unit for formal knowledge, so a sane system keeps three layers: immutable raw traces for audit; per-run analysis recording what happened and the draft lesson; and comparison, clustering and induction across many similar traces into forward-facing Markdown knowledge documents ai-agent-book-9. What the formal document states is applicable situations, recommended strategy, forbidden actions, exception conditions, evidence sources, and when it was last validated — not a retelling of one particular task. This is the same shape as "User as Code": append facts to an immutable log first, then rebuild the structured model periodically. Save the evidence first, generate mutable knowledge offline — separating recording from sorting means a single accidental success or network failure cannot change agent behavior immediately, and the system gets the chance to see several outcomes before it sees a pattern.
Transferable content comes from contrast
An experience document is not a trace summary. What is worth distilling is the result of contrast: what successful traces of the same family did, what the failing ones lacked, which environment versions a strategy worked in, and which preconditions broke it ai-agent-book-9. The full pipeline has five steps: store the immutable trace and the environment outcome; generate a structured analysis of the single run; aggregate by task family and build, for each draft lesson, an evidence table of which traces support it and which contradict it; write to the formal document only once the support threshold is met; and finally test transfer on new tasks that took no part in the distillation. The GAIA and AWorld examples draw the line between the old and the new practice: the old one generated a strategy summary the moment a task succeeded and vectorized it into the store; the rigorous one first has the environment verifier label success, partial success and failure, then compares multiple paths in the same family — successful traces contribute strategy drafts, failing traces contribute exclusionary knowledge, and partial successes show which segment worked and which one is still broken ai-agent-book-9. Reflexion-style natural language reflection may take part in generating drafts, but reflection is not evidence: only what matches the environment outcome, gains cross-trace support, and shows positive transfer on new tasks qualifies for the formal document.
Sleep learning: sorting scheduled into idle time
"Sleep learning" is the cognitive analogy for offline consolidation; it does not require running at night. The online agent's first duty is to finish the current task and append immutable evidence; a background learning process, during idle periods or when gating conditions are met, batch-reads new experiences, compares old and new, merges duplicates, resolves conflicts, proposes updates and runs regressions ai-agent-book-9. A typical cycle has five steps: trigger (time interval, number of new traces, or error frequency crosses a threshold while no high-priority task is running); orient (read the formal knowledge, prompts, and Skill catalog with versions, and establish the unmodifiable boundary); collect and consolidate (find new signals in already-evaluated traces, preferring local patches); validate and approve (evaluate on transfer, held-out and safety sets, with high-risk writes waiting for human approval); prune and index (mark capabilities unused for a long time or overturned by new evidence as expired, archive or delete them, keeping provenance and rollback versions). Separation has an economic argument too: sorting can run in larger batches on cheaper models.
Do not confuse it with memory
Four Kinds of State, Four Lifecycles answers "where does this go and how long does it live"; the distinction here runs along another axis: user memory accumulates "what the user and the world are like", experience learning accumulates "how to act under which conditions" — the first makes the agent remember more, only the second makes it go from clever to competent ai-agent-book-9. One self-check question: inside your "memory system", is there a single experience that changed how the system behaved afterwards, after it was written? If every write only goes in and never out, only stores and never proves, that is not a learning loop — it is an archive that gets more expensive every year.