Conversation Compaction

Long chats quietly drop content. What drops first is usually the constraints set early on.

Conversations keep growing, but the window is finite. So any long-conversation product eventually faces the same question: when it doesn't fit anymore, what do you throw away? context-window-wiki

Symptoms you'll recognize:

• Halfway through a long conversation, it suddenly forgets a rule set at the start.
• It garbles a specific number you discussed earlier.
• You want to reproduce a problem but have no idea what's left in the context.

Why compress at all

Two reasons, and the second is the one people underestimate.

• It doesn't fit. The window has a hard limit; beyond it you have to do something.
• It's expensive. This is the real driver for most products. The whole context is resent and rebilled every round — the sentence in round ten may be ten words, but that call's input is the sum of everything before it. The relationship between conversation length and cumulative cost is close to quadratic: the longer you chat, the steeper it climbs.

Agent scenarios are the clearest case. Every tool result piles into the context, and after dozens of rounds the history alone can fill the window.

Three approaches

• Truncation. Drop the earliest rounds, keep the most recent. Simplest to implement, and it's the default behavior of many frameworks. The problem is also the most direct: the earliest rounds usually contain the most important stuff — the original task description, the user's background, the constraints you both confirmed. Once they're gone, the model starts to drift.
• Summarization. Hand the first N rounds to the model, condense them into a short summary, and replace the original with it. Better than truncation — the main thread survives. But a summary is lossy by nature: specific numbers, exact wording, and options you ruled out tend to evaporate. And the summary itself is model-generated, so it can be wrong — once it is, everything after it builds on a false premise.
• Layered retention. The more solid approach: handle content by importance. Keep hard constraints verbatim — explicit rules, key parameters, decisions that can't change, not a character altered. Summarize the middle process — discussions, attempts, rejected plans — into one block. Keep the most recent rounds verbatim, because the newest context needs precision most.

It's more work, but the stability difference is large. The core idea: what must be remembered stays exact; what can be fuzzy gets fuzzy.

What usually gets dropped

The sentence to remember from this section: compression most often drops the constraints established early on.

The reason is practical. Constraints usually get set at the start of a conversation — "reply in Chinese," "no more than three paragraphs," "only use the material I gave you." They're said once and never repeated. In a summary they don't look like "main content," so they get summarized away.

The result is the classic symptom: mid-conversation it starts violating the rules agreed at the beginning. Users think "it got dumber." Actually the rule is no longer in the context.

Second most lost: precise numbers and identifiers — order IDs, amounts, version numbers, names. Summaries tend to generalize: "discussed refund issues for a few orders" — which ones? Gone.

Third: negative information. "We decided not to do option A." A summary tends to simplify it to "discussed option A," and later the model may pick A back up.

Four practices

• Store constraints separately, outside the compression flow. Pull the user's hard constraints into a fixed zone (usually right after the system prompt) and never compress them. This one step fixes most drift.
• Log the compression. When it happened, how many rounds were dropped, and what the summary said. Without this log, production problems are basically unreproducible — you don't even know what was in the context.
• Consider making it visible. If compression noticeably affects the experience, a notice ("earlier conversation summarized") beats having the user discover the amnesia by accident. Some products let users "pin" key information — manually adding to the never-compress zone.
• Ask whether you need such a long conversation at all. The most easily skipped and the most effective: many scenarios don't need an endless session. A clear "start a new task" entry point beats any compression strategy — cheaper and more predictable. Compression is a necessary patch, not the default good design.