Short-term memory

What is short-term memory?

Short-term memory is the working context an AI agent holds inside one active session: the last several turns of dialogue, the customer record it fetched, the tool outputs it received, and the system instructions that frame the whole exchange. It exists for the duration of the conversation and then clears.

That working set is bounded. Every model has a fixed ceiling measured in tokens, and once a conversation plus its retrieved documents crosses that ceiling, something has to be dropped, summarized, or reordered. Support conversations reach it faster than most teams expect, because attachments and order histories consume it too.

How short-term memory works

Short-term memory is assembled fresh on every turn. The runtime builds a prompt, sends it, and discards the assembly afterwards; nothing persists inside the model itself between calls. Four layers usually go into that assembly.

The first is the system layer: instructions, persona, policy constraints, and the tool definitions the agent is allowed to call. The second is the dialogue buffer, the recent user and assistant turns held word for word. The third is retrieved context, the knowledge passages and account records pulled in for this specific question. The fourth is tool state, meaning the result of the refund lookup or the shipment query the agent just ran.

All four compete for one budget, because the context window is the hard ceiling every layer is charged against, and that ceiling is counted in AI tokens. When the assembled total exceeds it, the AI agent framework applies an eviction policy: truncate the oldest turns, compress them into a running summary, or drop the retrieved passages that scored lowest. That policy decides what the agent still knows on turn twenty.

Types of short-term memory

Most production agents run several of these at once, each with a different eviction behaviour.

• Verbatim turn buffer: Recent messages held word for word so pronouns, corrections, and follow-ups resolve against exactly what the customer said, though it consumes budget fastest.

• Rolling summary: A compressed account of the conversation that replaces older turns once they fall out of the buffer, at the cost of specific detail.

• Scratchpad state: Intermediate reasoning, plan steps, and tool results the agent writes for itself mid-task, usually invisible to the customer reading the reply.

• Pinned session slots: Structured values such as order ID, account tier, or language, held in fixed fields so they survive every eviction pass.

• Retrieved context: Passages and records fetched for the current turn, kept only as long as that turn needs them, which makes stale retrieval a recurring failure.

Short-term memory vs long-term memory vs context window

These three get used interchangeably in architecture discussions, and the conflation hides which one you can actually change. Long-term memory stores durable facts about a customer across separate conversations, surviving after the session ends. A context window defines the maximum token capacity a model can attend to on a single call, fixed by the model itself. Short-term memory manages what occupies that capacity during one live session, deciding which turns, records, and summaries earn their place. The window is the container; short-term memory is the policy that fills it.

If the problem is that a customer repeats themselves inside one conversation, fix short-term memory. If they repeat themselves across weeks, you need long-term memory. If both are configured well and answers still truncate, the window is your ceiling.

Why short-term memory matters for customer experience

When short-term memory is shallow, the agent treats every message as the first one. The customer gives an order number, then gets asked for it again two turns later, and the conversation restarts from zero each time. Repetition is the visible symptom. The deeper cost is that the agent cannot reason across turns at all, so it never connects "my card was declined" with the address change mentioned three messages earlier. A working multi-turn conversation exists only because this layer holds.

The tradeoff is real and per-turn. The entire assembled context is re-read on every single call, so holding more history adds latency and cost to each reply. A team that keeps everything is paying to re-read turn one's greeting at turn forty, on every turn in between.

How is short-term memory measured?

Short-term memory produces no single score, so teams instrument four observable quantities: context utilization (tokens consumed against the ceiling), retention accuracy (how often a fact given on turn two is still used correctly on turn fifteen), eviction loss (what a summary dropped that later turns needed), and time to first token, which climbs as assembled context grows.

Retention accuracy is the one worth building a harness for. Script a fixed set of conversations, plant a fact early, ask a question that depends on it late, and chart the pass rate against turn depth. The curve tells you where your eviction policy starts failing.

The ceiling itself keeps moving. Frontier model context lengths tracked in the Stanford HAI AI Index (2025) run from a few thousand tokens in earlier generations to the high hundreds of thousands and past a million in some releases, which changes what gets evicted without removing the need to evict.

How AI agents manage short-term memory

Modern agents run a memory manager between the conversation and the model. On each turn it scores what is available, pins the fields the task cannot proceed without, keeps the most recent turns verbatim, and compresses everything older into a running summary. Retrieval happens on demand, so a policy document enters the assembly for the turn that needs it and leaves afterwards.

Two mechanisms decide quality. The first is the summarization prompt, which determines whether identifiers, dates, and stated preferences survive compression or get smoothed into prose. The second is pinning: any value that must never be evicted is stored as a slot rather than trusted to the buffer.

The consequence lands at escalation. When the agent hands off, the pinned slots and running summary become the context a human inherits, which is the difference between a clean transfer and a customer explaining everything twice. Teams designing for full context handoff treat that payload as a deliverable.

Implementing short-term memory

Start with coverage: list the values a session cannot lose (customer identifier, order reference, verification status, language) and pin every one of them explicitly. Anything left to the buffer will eventually be evicted by a long conversation.

Integration surface comes next. Session facts arrive from a CRM, an order system, and a billing platform, and each lookup you can defer until the turn that needs it is budget you keep for dialogue.

Governance means naming who owns the eviction policy and how long session transcripts persist after the conversation closes. Buffers hold personal data, so GDPR questions about deletion arrive early, and enterprise buyers ask how session handling is evidenced under SOC 2 Type II.

The constraint that bites hardest is latency on voice. Assembled context is re-read before the first token is produced, and a spoken reply that arrives a second late reads as a broken call.

Short-term memory and the agent memory stack

Short-term memory is one tier of AI agent memory, the broader architecture that also covers durable customer facts and retrieved knowledge. Reading them together clarifies which tier a given failure belongs to, because a customer who repeats themselves within a session and a customer unrecognized after a week have different causes.

The bridge between tiers is compression. Conversation summarization is what turns an expiring buffer into a durable record: the same mechanism that keeps a long session inside its budget also produces the artifact stored after it ends.

What does short-term memory mean in plain terms?

Think of short-term memory as the notepad a phone agent fills during a single call: the name, the order number, what was already tried, scribbled down and thrown away when the call ends. Nothing on it survives to the next caller, and that is the point.

Without the notepad, the agent asks for the order number, forgets it while checking the warehouse, and asks again. The customer hears someone who is not listening, and hangs up before anything is resolved.

The tradeoff is that a bigger notepad is slower to read. Every fact you keep on it gets re-read before the next sentence is spoken, so keeping everything makes the agent thorough and sluggish at once. Choosing what to throw away is the actual design work.

Common short-term memory mistakes

Pinning nothing is the first pattern. Teams trust the buffer to hold the order number, then discover it was evicted because it arrived on turn two of a thirty-turn conversation. Identifiers belong in fixed slots that eviction cannot touch.

Summarizing for humans is the second. A summary written to read nicely drops account numbers, exact dates, and the precise wording of a customer's request, and the agent then reasons from a version of the conversation that no longer contains the facts.

Confusing storage with memory is the third. Logging the full transcript to a database feels like persistence, but if nothing re-injects it into the next prompt, the agent still starts blind.

The fourth is dropping the buffer at handoff. Context that existed a second earlier vanishes when a human takes over, which is why human fallback design treats the transfer payload as part of the agent, not as an afterthought.