An Evolution Loop Needs Three Brakes
Proving the system got stronger is not enough; limit who may change what: evidence apart from instructions, candidates quarantined, safety mechanisms closed to self-edit.
Self-evolution turns a single mistake into a fork: caught in time it becomes an asset, unrestrained it becomes a standing risk. A prompt injection summarized into an experience entry takes effect across sessions; a malicious package wrapped as a tool spreads from one sandboxed run to every later task; a flawed verifier keeps approving versions that look like progress and are in fact regression ai-agent-book-9. So an evolving system has to answer a second question besides "did it get stronger": who may change what, and where the input comes from. The three boundaries are one answer to that question.
First brake: evidence apart from instructions
Raw web pages, tool outputs, and their LLM summaries are all untrusted evidence: they must not be executed as instructions, and must not be folded directly into long-lived capabilities such as Skills ai-agent-book-9. The key realization is that an LLM summary is a transformation for readability, not a sanitization process that makes input harmless — injected content phrased more fluently is still injected content. The engineering form is to extract claims, source position and collection time against a fixed schema, keep the original text and its provenance alongside, and never execute the extracted string as an instruction. Two details are commonly skipped: a confidence figure produced by the model is an untested estimate and cannot serve as an approval threshold; and content waiting to be published first passes deterministic schema, allow-list and provenance checks, then is submitted as a versioned pull request reviewed against the original evidence by a reviewer independent of the generator, with manual approval added before a high-risk Skill ships.
Second brake: candidates apart from shipped capabilities
New knowledge, prompts, Skills, code and parameters enter a holding area after generation — an area that cannot serve real traffic ai-agent-book-9. The safety checks finish here: new code and external dependencies go through sandbox, permission checks, supply-chain scanning and behavioral tests, and only become shipped capabilities afterwards. The support threshold for experience documents in Saving Experience Is Not Learning From It — write to the formal document only once the threshold is met — is this same quarantine expressed on the distillation side. The quarantine also pays off along time: an accidental success, a network failure, a malicious input should none of them rewrite system behavior the same day. Delete "effective immediately" from an evolution system and most incidents lose their propagation path.
Third brake: safety mechanisms cannot self-modify
The business agent may change prompts, Skills, the knowledge base and tools, but not the verifiers that approve its own updates, the test cases, the release gates, the audit log, or the stable-version backup ai-agent-book-9. The reasoning is blunt enough to be frightening: otherwise an agent only needs to lower a test threshold or delete a failing case to disguise regression as progress. Two experiments pin this down as mechanism rather than principle: a confirmation gate for high-risk operations triggered by user feedback must pass validation on both a dangerous-boundary set and a normal hold-out set — it has to stop unconfirmed high-risk calls without blocking ordinary tasks — and the safety gate itself cannot be modified by the proposal ai-agent-book-9. The agent generating proposals has no right to touch safety tests or approval rules; a proposal being refused by the safety gate is precisely the independent verifier and the unmodifiable root of trust working — which matters far more than the proposal looking reasonable. Same logic as the data layer of the three guardrail layers: a real guarantee must come from a layer the thing being modified cannot reach.
Humans stay in the loop
Beyond the three brakes, the residual risk is handled by position: in the validate-and-approve step of sleep learning, high-risk writes wait for human approval; in pruning and indexing, provenance and rollback versions are kept ai-agent-book-9. Rolling back is not shameful — an evolving system with no rollback is what should be shameful. Sandboxes, Credentials, and Human Boundaries governs who may do what inside a single run; this article governs who may change what across versions. With both layers standing, "self-improvement" is an engineering word rather than a marketing one.