Conversation Compaction

When a conversation outgrows the window something gets thrown away, and the earliest constraints go first. Anything that must survive belongs outside the conversation.

Conversations keep growing, the window does not. So every long-conversation product eventually answers the same question: when it no longer fits, what do you throw away. The default answer is "the earliest turns" — and the earliest turns usually hold the rules you set. context-window-wiki

After reading, you should be able to answer:

• Which three kinds of content does compaction drop first?
• When should you compact deliberately rather than let it happen?
• Why must anything that survives live outside the conversation?
• What is cheaper and more reliable than any compaction strategy?

Two triggers, and the bill is the one people forget

It does not fit. The window has a hard ceiling, and past it you have to do something. Nobody escapes this one.

It is expensive. For most products this is the real driver. The whole context gets sent again and billed again on every round — the sentence in round ten may itself be ten words, yet that call's input is the sum of everything ahead of it. The further a conversation runs, the more the cumulative cost approaches quadratic growth: double the length and the bill more than doubles.

Agent scenarios make it most obvious. Every tool call piles on another block of results, and after a few dozen rounds the history alone is enough to fill the window — both triggers arriving at once.

What separates the three approaches is what each one loses

One: truncation. Keep the most recent rounds and discard the earliest. Least work to implement, and what many frameworks do by default. What it loses is the material the opening rounds carried — how the task was originally described, what background the user supplied, which constraints both sides settled. Once those are gone, the model starts to drift.

Two: summary replacement. The earlier rounds stop being kept verbatim; a model condenses them into a short block that stands in for the original. Better than truncation — the thread survives. But it is lossy by nature: exact numbers, wording that had to be exact, and the options ruled out along the way tend to vanish during condensation. The summary is model-generated too, so it can get things wrong — and when it does, everything downstream rests on a false premise.

Three: layered retention. Content is handled according to importance: hard constraints verbatim (rules the user laid down, key parameters, decisions already settled and not to be reopened — not a character changed); the middle process summarized (discussion, attempts, rejected plans, condensed into one block); the most recent rounds verbatim (the newest context needs precision most). More work, but a clear gain in stability.

What actually separates the three is not the technique but which category you are willing to lose. That is why the third is the steadiest: it splits "must never be lost" from "acceptable to lose" instead of compressing everything as one mass.

Compact on purpose, or let it happen to you

Compaction does not have to wait for the window to fill. Two timings, and the difference matters:

Passive compaction is the framework noticing it does not fit and acting on its own. You have no say; what gets dropped follows someone else's default rule.

Deliberate compaction happens at a node you chose: a phase ended, a task was delivered, the user clearly switched topics. The advantage is that you decide what goes — and you can move what must survive before it does.

The test is simple: if this conversation no longer needs the early trial and error, compact now; if it does, compacting creates a problem for you. Compaction itself costs a model call, so compacting too often is its own waste.

The three categories that vanish most predictably

One: constraints set early. This category is lost more than any other. Constraints get settled at the very start — never print internal system codenames, keep every paragraph under two lines, cite only the inspection report the user uploaded. Said once, and nobody brings them up again. By the time a summary is written, those sentences do not read like "body content," so they get folded away. The symptom is the familiar one: halfway through, it stops honoring the rules agreed at the outset. The user assumes it got dumber; the truth is that the rule is no longer in the context.

Two: exact numbers and identifiers. Ticket IDs, reimbursement caps, rework batch numbers, the inspector's name. Summaries lean toward the general — "discussed a few reimbursement cases" — and which ones those were is unrecoverable.

Three: negative information. "We decided not to reapply for an extension" easily becomes "discussed reapplying for an extension," and the model may later treat it as work to be done. Of all meanings, negation is the easiest to summarize away, because in the original it occupies a single word.

All three share a trait: they are the kind of information said once and never needed again — and to a compressor, never repeated means never important.

Three rules for the flow itself

One: keep load-bearing constraints out of the compaction flow. Pull the user's hard constraints out on their own and give them a fixed position (usually immediately after the system prompt), never subject to compaction. Most drift disappears at this step, and it is simpler than any summarization strategy.

Two: log every compaction. When each one ran, how many rounds it removed, what the summary said. Without this log, production problems are basically unreproducible — you cannot even say what was in the context at the time.

Three: consider making it visible. If compaction noticeably affects the experience, a notice ("earlier conversation summarized") beats a user running into the amnesia with no warning. Some products let users pin key information, which is manually adding to the never-compact zone.

Two questions to ask before any of this

First: does this conversation need to be this long? The most-skipped step, and the most effective. Many scenarios do not need a single endless session — offering a clear entry point for a new task beats any compaction strategy, costs less, and behaves more predictably. Compaction is a necessary patch, not the default good design.

Second: why does this information have to live in the conversation at all? If a piece of information must survive, it should not exist only in the conversation history. A conversation is volatile storage — it gets compacted, truncated, and summarized away. Anything that can be written into structure is more reliable than the conversation: user preferences become settings, task state goes into a business database, cross-session facts go into long-term memory, phase results become a handoff artifact.

Those two questions turn "how do we compact better" into "what should never have depended on the conversation in the first place." The first is a trade-off inside a lossy process; the second keeps information out of that process entirely. So the order is: move out what belongs outside first, then decide how to compact the rest. For what has to persist across sessions, what it costs to make it remember covers storing and retrieving it; for passing state between long rounds, Layered Loading and Handoffs is the system-side answer.