Context engineering

What is context engineering?

Context engineering is the practice of deciding what information reaches a language model at inference time, in what order, and in what format. It covers retrieval, selection, compression, tool output, conversation history, and the system instructions that frame all of it.

The discipline exists because model windows are finite and every token inside one costs money and latency. A support agent answering a single billing question may need three policy passages, the customer's last two tickets, and a live subscription record, all assembled in under a second.

How context engineering works

A context engineering pipeline assembles five layers for every single request: instructions, retrieved knowledge, conversation state, tool output, and the budget that reconciles them.

Instructions come first and change slowest: role, tone, refusal rules, and output format. Retrieved knowledge arrives next, usually through retrieval-augmented generation, which searches a knowledge store at answer time and returns the passages that match the question. Conversation state carries the turns so far, compressed once the thread grows long. Tool output is whatever a function call returns: an order status, a subscription tier, an open ticket.

The budget layer is where the engineering actually happens. All four upstream layers compete for one context window, and every passage costs AI tokens that add latency and price to the call. The assembler ranks candidates, truncates the weakest, drops what will not fit, and tags each surviving passage with its source so the answer can be checked afterwards.

Types of context engineering

• Retrieval context: Passages pulled from documentation, help centers, and resolved tickets at answer time, ranked by relevance to the current question.

• State context: The conversation so far plus durable customer memory, usually summarized once the thread outgrows its share of the budget.

• Tool context: Structured output returned by function calls, such as an order record or an entitlement check, injected as fresh facts.

• Instructional context: Role, tone, escalation rules, and output format, the slowest-changing layer and the one most often edited without review.

• Compression: Summarization, deduplication, and truncation that fit the other four layers into the window, at the cost of detail nobody can recover later.

Context engineering vs prompt engineering vs RAG vs fine-tuning

These four get used as synonyms in vendor decks, and the confusion costs teams months of the wrong work. Prompt engineering shapes the standing instruction a model reads on every request. Retrieval-augmented generation fetches the passages that answer this particular question. Fine-tuning changes the model's own weights so behavior persists with no input at all. Context engineering governs the whole assembly: what gets fetched, what survives the budget, and in what order the model sees it. The four operate at different frequencies, which is the cleanest way to tell them apart.

If answers are wrong about this customer or this order, the work is context engineering. If they are wrong in tone, format, or willingness to refuse, fix the instruction layer first, because assembling more material will never correct an instruction problem.

Why context engineering matters for customer experience

A support answer can only be as good as the material handed to the model. When nobody engineers the context, the failure is quiet: the agent replies fluently from a stale policy passage, or misses the customer's open refund case because nothing pulled ticket history into the window. The customer experiences an assistant that sounds informed and is wrong about their specific situation, which costs more trust than a plain refusal would.

The tradeoff is unavoidable. Every extra passage improves the odds that the answer sits somewhere in the window, and it simultaneously raises token cost, adds latency, and gives the model another source to anchor on. Teams handling complex support inquiries generally find that fewer, tightly scoped passages beat broad retrieval, because a narrow window forces the ranking to be right.

How is context engineering measured?

Measurement happens inside the assembly step, before any answer exists. Retrieval precision asks how often the passages handed to the model actually contain the answer, scored against a labeled question set someone graded by hand. Context utilization asks what share of the assembled window the final answer drew on, which exposes padding. Token cost per resolved conversation and time to first token convert the same assembly into money and waiting.

Cost has a public reference point. The Stanford HAI AI Index reported that the inference cost of running a model at a fixed capability level fell more than 280-fold between November 2022 and October 2024, which is why context budgets that were uneconomic in early deployments are now ordinary.

Answer quality has no equivalent published reference, so run the comparison against your own baseline: the same question set, one variable changed in the assembly, graded the same way each time.

How AI agents change context engineering

The mechanism changes first. An AI agent decides at runtime what it needs, so the context stops being a fixed template a person wrote in advance. The agent issues a retrieval query, reads the result, judges whether it is sufficient, calls a tool, and reassembles the window before writing anything. Each of those steps alters what the model sees on the next pass.

That moves the work from authoring text to designing the loop and its limits: how many retrieval rounds are permitted, what happens when a tool call times out, and which facts must be present before the agent is allowed to answer at all.

The consequence shows up at escalation. When the agent hands off, the assembled context is the transcript a human inherits, which is why context preservation at handoff belongs to the same discipline rather than to the ticketing tool.

What to look for in a context engineering stack

Source coverage is the first axis: how many systems the assembler can reach, and whether it can pull a document passage and a live account record inside one request. Integration surface is the second, since retrieval and tool calls have to run against the help desk, CRM, and billing systems you already maintain.

Governance decides whether any of it survives an incident review. The instruction layer needs prompt versioning so a past answer traces back to the exact instruction behind it. Regulated buyers usually raise ISO 42001 for AI management systems and GDPR where personal fields enter a prompt, and the answerable form of both questions is whether you can reconstruct, for one past conversation, which customer data entered the window and who approved that.

The constraint teams underestimate is the latency budget. Retrieval, reranking, and tool calls all run before the first token appears, so every extra hop is paid in silence while the customer watches a typing indicator.

Context engineering and grounding

Context engineering and AI grounding describe two halves of one guarantee: assembly decides which sources reach the model, and grounding is the requirement that the answer stay traceable back to them. An assembler that discards provenance makes grounding unverifiable even when the retrieved passage was perfectly correct.

The relationship to retrieval is narrower than most teams assume, and the distinction between grounding and RAG is the useful place to start, because retrieval supplies only one layer while context engineering also governs tool output, state, and everything else competing for the same window.

What does context engineering mean in plain terms?

Think of context engineering as packing a briefcase for someone walking into a meeting they had no time to prepare for. They will speak confidently about whatever is in the folder and know nothing about what you left on the desk. The whole job is choosing the pages that matter for this meeting.

Skip the packing step and the model answers from whatever happened to be nearest: a marketing page, a two-year-old policy, a half-remembered earlier turn. It will still sound certain, because fluency has never depended on having the right documents.

The tradeoff you accept is that packing costs time and judgement on every single request. A fixed instruction is written once and reviewed occasionally; assembled context is rebuilt for every customer and every turn, and someone has to own the rules that decide what gets in.

Common context engineering mistakes

Four patterns account for most context engineering failures.

Filling the window because the capacity exists is the first. A large window tempts teams to pass everything available, which raises cost and latency on every call and gives the ranker more chances to surface a passage that merely looks relevant.

Treating retrieval as a solved component is the second. Semantic search returns something for every query, including queries it has no answer for, so an empty result set almost never appears to signal that a knowledge gap exists.

Letting state grow without a policy is the third. Long conversations get summarized, the summaries get summarized again, and the customer's original constraint disappears two turns before the agent contradicts it.

Owning the pipeline while nobody owns the content is the fourth. Assembly cannot repair a knowledge base that contradicts itself; it only decides which of the two contradictory passages the model reads today.