How Long-Term Memory Works

Extraction, conflicts, injection. What it costs to make a model recognize you.

"An AI assistant that remembers your preferences" is an attractive selling point. Get one thing straight first: the model has no memory across sessions. Every "it remembers me" moment is your system storing and retrieving on its behalf.

Signs you'll recognize:

• Users complain, "I already told you this — why are you asking again?"
• After shipping memory, it latches onto random trivia.
• A user changes a preference, but it keeps following the old one.

The three steps

Extract. When a conversation ends (or even mid-way), a model call decides which facts are worth keeping: "I usually work in Python," "Our company is in Shenzhen," "Don't email me reminders."

Store. New memories are written to the store, and conflicts are handled. Three months ago the user said "I live in Beijing"; now they say "I moved to Shanghai." Those two can't coexist — update, don't append.

Inject. At the start of the next session, pull a few relevant memories into the system prompt or the opening context. The model sees them and behaves as if it remembers you.

Step three is worth spelling out: it's the same mechanism as RAG — retrieve relevant content, stuff it into context. The only difference is the retrieval target: facts about this user instead of document fragments. Research like MemGPT makes this loop explicit: the model needs an "operating system" to manage memory in layers, which is also the most common blueprint for production memory features. memgpt

Where it gets hard

Of the three steps, the first two are the real problems.

Extraction has no objective standard for "worth keeping." Keep too much and the store fills with trivia — an offhand joke, a one-time request — all treated as long-term preferences. Every later session injects them into context, costing money and muddying the model's judgment. Keep too little and there's no point. The workable approach is to constrain categories: only keep a few well-defined kinds of information, such as long-term preferences, identity facts, and explicit prohibitions. Ignore everything else. A vague "remember what matters" almost always drifts.

Conflict handling is hard because you must decide which fact wins. Usually the newer one. But "use English this time" is temporary and shouldn't overwrite "I usually work in Chinese." Telling temporary from permanent is where this step fails most.

There's also a hidden problem: memories are extracted by a model, and models make mistakes. A wrong memory, once stored, quietly affects every later conversation — and users can't see what the system recorded about them.

The cost

Memory costs more than it looks. Add up three bills:

• An extra extraction call per conversation. Invisible to users, paid by you.
• Injected memories occupy the window and get resent every round. More memories means a larger fixed cost.
• Privacy and compliance. You are continuously storing personal information, which touches your privacy policy, data retention, and users' right to deletion. Run it past legal before building.

One product requirement is non-negotiable: users must be able to see and delete their memory. A page listing "here's what the AI knows about you," with per-item deletion. It's compliance and trust in one.

Should you build it

Practical advice: skip general memory first; start with a few explicit fields.

Most products don't need a full memory system. Language preference, industry, preferred formats — these can be explicit settings the user fills in. Predictable, editable, zero extraction error, zero privacy debate, and nearly free.

The test: if the user would willingly tell you this information, make it a setting. Don't make the AI guess.

Automatic memory earns its cost only for things users won't volunteer but that genuinely affect the experience — work habits, recurring mistakes, preferred phrasing. Those justify a full extraction-and-conflict pipeline.

One more boundary: memory stores facts about the user, not their full chat history. Logs are a different thing with different storage, retention, and compliance rules. Don't mix them.