Token limit

What is a token limit?

A token limit is the maximum number of tokens a model will accept or produce in a single request, counted across the system prompt, the conversation so far, any retrieved documents, and the reply itself. Exceed it and the request is truncated or rejected outright, depending on the provider.

Limits are set by the model, the serving platform, and the application layer on top, and the tightest one wins. A rough working figure: one token averages about four characters of English, so a thousand-word support policy consumes roughly 1,300 tokens before the customer has said anything.

How a token limit works

Every request passes through four stages, and the limit binds at the third. First, a tokenizer splits text into subword pieces: AI tokens are the unit, so "refund" may be one token while an order ID becomes six. Second, the application assembles the request, concatenating system instructions, tool definitions, retrieved passages, prior turns, and the new message into one string. Third, the serving layer counts the assembled tokens against the context window of the model and against any per-request cap the platform enforces on top of it. Fourth, generation runs, and each output token is appended to the same running total.

Two properties follow from this. Cost and latency scale with the count, because attention compares tokens against one another as the sequence grows. Input also competes with output: a large language model asked to reason over a long history has less room left to answer in.

Types of token limits

• Context window limit: the model's architectural ceiling on prompt and completion combined, fixed by how the model was trained and served.

• Max output tokens: a per-request cap on the completion alone, set by the caller, which silently cuts a long answer mid-sentence when it binds.

• Rate limits in tokens per minute: an account-level throughput quota governing how many tokens you may send across all concurrent requests in a rolling window.

• Application budgets: caps a team imposes on itself for retrieved passages or history length, usually the only limit anyone can tune this week.

• Embedding input limits: the separate and usually much smaller ceiling on how much text an embedding model will accept per chunk.

Token limit vs context window vs rate limit vs output cap

These four get used as synonyms in the same sentence, and the substitution hides which one is actually blocking a request. A context window defines how much a model can hold in view at once. A rate limit defines how many tokens an account may push through per minute. A max output cap defines how long a single completion may run. A token limit is the general name for any of these ceilings, which is why the useful question is always which one bound first.

Instrument all four before tuning any of them: log the assembled input count, the completion count, and the error class per failure. Context window trouble looks like forgetting, rate limit trouble looks like flakiness under load, and an output cap looks like an unfinished sentence.

Why token limits matter for customer experience

A token limit fails quietly, which is what makes it a customer experience problem. When an assembled request exceeds the ceiling, most systems drop the oldest turns or the lowest-ranked passages, so the model still answers, fluently, from an incomplete picture. The customer sees an agent that forgot the account number they gave two turns ago, or one that quotes a refund window from the policy paragraph that survived while the exception clause was cut.

A dropped constraint is one of the cheapest routes to a hallucination, because nothing in the output signals that anything went missing. The tradeoff is real: buying headroom costs money and latency on every request, including the majority that never needed the extra room. Teams that push everything into the prompt to be safe pay for that headroom on thousands of short questions a day.

How is a token limit measured?

Serving throughput at fixed sequence lengths does have public benchmarks: MLPerf Inference reports tokens per second for defined input and output lengths on defined hardware. That taxonomy belongs to hardware comparison, and its sequence lengths are chosen for comparability across systems, so its figures say nothing about whether your support prompts fit. There is no standards-body target token count a support team is expected to hit.

What you measure locally is a distribution. Instrument the tokenizer at request time and log the counts per turn for system and tool definitions, retrieved context, conversation history, and completion. Then read the tail, since the average is comfortable on almost every deployment and the worst percentiles are where truncation lives. The measure function of the NIST AI Risk Management Framework describes this pattern: define the metric, instrument the system, and keep tracking it after release.

How AI agents change token limits

A single-turn assistant sends one question and gets one answer, so the count is small and stable. Agentic AI changes the shape: a planner writes a step, calls a tool, reads the result back into the prompt, and repeats. Every tool result, every intermediate reasoning step, and every schema for the next call is appended to the same request, so a task that takes eight tool calls carries the transcript of the first seven into the eighth. Cumulative consumption grows with the square of the conversation length in the worst case, because each turn re-reads everything before it.

The consequence for support teams is that the limit binds on exactly the hardest cases: the multi-system refund, the escalation with a long history, the ticket that needed four lookups. Designs for agentic support workflows handle this by summarizing completed steps and discarding raw tool payloads once the useful field has been extracted.

How to reduce token usage under a token limit

Four levers move the count, in the order they usually pay off. Retrieval precision comes first: returning three tightly scoped passages beats returning twelve long articles, and it improves the answer as well as the bill. Summarization comes second, replacing resolved turns with a short state record that keeps the account number and the decision. Third, prune tool schemas so only the tools reachable from the current step are described. Fourth, split the task, since two focused requests often fit where one combined request will fail.

Data protection bites here in a way teams miss. GDPR's data minimisation principle argues against pasting an entire customer record into a prompt when the agent needs two fields, and the smaller payload is also the cheaper one. The constraint that hurts most operationally is that truncation leaves no error to alert on, so the counting work has to be budgeted before the tuning work.

Token limits and conversation memory

The token limit is the physical constraint behind every memory decision in a multi-turn conversation. Retaining full history is the simplest design and the first to break, so teams keep a rolling window of recent turns alongside a compact record of facts already established. Retrieval is the other half: a knowledge base is queried per turn precisely so the whole corpus never sits in the prompt, and the quality of that query decides how much relevant policy each token buys.

What does a token limit mean in plain terms?

Think of a token limit as the size of a desk. Everything the model uses has to be laid out on it at once: the instructions, the notes from earlier in the conversation, the pages pulled from the manual, and the space left to write the reply. When the desk fills, something gets pushed off the edge, and the model keeps writing as though it were still there.

Without a limit you could hand over the entire company handbook with every question and never think about it again. With one, someone has to decide each time which pages are worth the space, and that decision is made either by code your team wrote or by a default it inherited.

The tradeoff is between memory and money. A bigger desk remembers more and costs more on every request, including the thousands of simple ones that would have fit on a small desk.

Common token limit mistakes

Three patterns account for most incidents.

Estimating tokens by word count is the first. English averages roughly four characters per token, but JSON payloads, order IDs, base64 blobs, and non-Latin scripts tokenize far more densely, so a request sized from a word count can be double the estimate once real ticket data flows through it.

Treating truncation as a display problem is the second. When history is dropped, the model produces a complete, confident answer from partial input, and no error is raised anywhere. Teams catch this early by logging the assembled count and alerting on drops, the same discipline behind guardrails for support automation.

Buying headroom as a fix is the third. Moving to a larger context window clears today's error and raises the per-request cost of every conversation, while the underlying mechanism, an assembly step that includes everything it can find, keeps growing until it reaches the new ceiling too.