Large language model (LLM)

What is a large language model?

A large language model is a neural network trained on very large collections of text to predict the next unit of language, and through that single objective it learns grammar, facts, reasoning patterns, and style well enough to answer questions and follow instructions.

GPT-4, Claude, Gemini, and Llama are all large language models. The label "large" refers to parameter counts in the billions and training corpora measured in trillions of words, a scale that separates these systems from the intent classifiers that preceded them in support tooling.

How a large language model works

Training runs in stages. Pretraining exposes the network to a broad text corpus with one objective: predict the next token given everything before it. The model has no database of facts, only weights that encode statistical regularities, which is why its knowledge is fluent and approximate at the same time.

Text enters and leaves as AI tokens, sub-word fragments that also define billing and latency. Attention layers let every token weigh every other token, and the span of text the model can consider at once is its context window. Everything outside that window is invisible to the model during a given call.

After pretraining come instruction tuning and preference alignment, which teach the model to follow directions and refuse unsafe requests. At inference time, production systems add retrieval: relevant passages are fetched and placed in the prompt so the answer rests on AI grounding rather than on recalled weights. This retrieval step is what turns a general model into a system that can answer questions about your refund policy.

Types of large language models

• Foundation models: Broad general-purpose models trained on mixed web, book, and code corpora, sold through an API and adapted at the prompt layer.

• Instruction-tuned models: Foundation models further trained on demonstrations and preference data so they follow directions reliably, which is what almost every product actually calls.

• Open-weight models: Models whose parameters are downloadable, allowing self-hosting and fine-tuning, at the cost of running the serving infrastructure yourself.

• Small language models: Compact models in the low billions of parameters, fast and cheap per call, capable on narrow tasks and brittle on open-ended reasoning.

• Multimodal models: Models that accept images, audio, or documents alongside text, useful when a customer sends a screenshot instead of a description.

Large language model vs NLU vs rule-based chatbot

The categories blur because all three sit behind a chat box and all three produce replies. Natural language understanding classifies an utterance into a fixed intent and extracts entities from it, returning structure rather than prose. A rule-based chatbot matches keywords or decision-tree branches and emits an author-written response verbatim. A large language model generates each response token by token from learned distributions, so it handles paraphrase, typos, and mixed intents in one turn without a script per case. The generative approach buys coverage and gives up the determinism the other two provide by construction.

Most support stacks use two of these three. Run a large language model where the customer's phrasing is unpredictable, keep deterministic rules on the steps where a wrong word creates legal or financial exposure, and use natural language understanding output when routing must be auditable.

Why large language models matter for customer experience

Before generative models, coverage was authored by hand. Every phrasing of "where is my order" needed a training utterance, every new policy needed a new branch, and every unanticipated question fell through to a queue. Support teams spent their maintenance budget on the taxonomy rather than on the answers.

A large language model collapses that work. One model handles paraphrase, misspellings, two questions in a single message, and over a hundred languages without a script for each case, so coverage stops being a function of how many utterances someone wrote last quarter.

The tradeoff is real and structural. A generative model will produce a plausible answer even when it has no supporting source, so the failure mode moves from "the bot did not understand" to "the bot answered confidently and wrongly." That shift is why AI hallucination became an operational concern rather than a research curiosity.

How is a large language model measured?

Model quality is measured with public benchmark suites that score reasoning, coding, factual recall, and instruction-following, then reported as accuracy percentages per suite. Independent tracking matters because vendor-published scores use vendor-chosen prompts and settings.

The Stanford HAI AI Index Report tracks these suites over time and documents how sharply the gap between top open-weight and top closed models has narrowed, from a difference of roughly eight percentage points in early 2024 to about 1.7 points by early 2025 on one of its tracked comparisons. Read benchmark scores as a relative ranking rather than as a prediction of behaviour on your own tickets.

Production measurement is separate work. Score real transcripts against a rubric, sample them continuously, and use LLM as a judge to make that scoring affordable at volume.

How AI agents change large language model deployment

An AI agent wraps the model in a loop: read the request, decide whether to retrieve, call a tool, inspect the result, and either answer or take the next action. The model contributes judgement about what to do next; the surrounding system supplies the facts, the permissions, and the stopping conditions.

That architecture changes which model properties matter. Instruction-following and structured-output reliability outrank raw eloquence, because a malformed tool call breaks the loop where an awkward sentence does not. Multi-step tasks also consume context quickly, so teams route by task shape using an LLM router, sending simple lookups to a small fast model and reserving the strongest model for reasoning-heavy turns.

The consequence for support is that the model stops being the product. What decides whether an agent resolves a ticket is the quality of the retrieval layer and the actions it is permitted to take, a pattern visible across AI support agents for chat and email.

What to look for in a large language model

Start with task fit measured on your own data. Assemble fifty to two hundred real customer messages with known correct answers and run candidates against them, because benchmark leaderboards do not capture your product vocabulary or your policy edge cases.

Then check the integration surface: reliable structured output, function calling, streaming, and a context window large enough for your retrieved passages plus conversation history. Governance follows. Ask where inference runs, whether your prompts and outputs are retained or used for training, and whether data residency can be pinned to a region. Regulated buyers commonly ask for SOC 2 Type II as evidence of operational controls and, where health data reaches the prompt, a signed BAA under HIPAA before any transcript touches the model.

The constraint that bites hardest here is deprecation. Providers retire model versions on their own schedule, and prompts tuned against a retired version rarely behave identically on its replacement, so version pinning and a regression suite are prerequisites.

Large language models and retrieval systems

A model's weights are frozen at training time, so anything that changed afterwards has to arrive in the prompt. That is why retrieval sits next to almost every production deployment: passages fetched from a knowledge source give the model current facts to work from, and AI grounding practices decide whether those passages actually constrain the answer.

Retrieval also interacts with cost. Every retrieved passage consumes AI tokens inside a finite context window, so teams that stuff twenty documents into a prompt pay more, wait longer, and often get worse answers than teams that retrieve three well-scoped ones.

What does a large language model mean in plain terms?

LLM stands for large language model. Think of it as an extraordinarily well-read autocomplete: it has absorbed enough text to continue almost any sentence sensibly, including sentences that happen to be questions about your return policy.

Ask it something covered by its training and it will usually be right. Ask it about your pricing change from last Tuesday and it will still answer, because continuing text is what it does, and a made-up answer looks exactly like a correct one from the inside. That is the counterfactual worth holding onto.

The tradeoff: you gain a system that understands almost any way a customer might phrase something, and you accept that it has no built-in sense of when it does not know. Every reliable deployment spends its engineering effort supplying that missing sense.

Common large language model mistakes

Treating the model as a knowledge store. Teams ask a general model about company-specific policy and are surprised by wrong answers. The mechanism is that weights encode training-time statistics with no lookup step, so without retrieval the model is generating a plausible policy.

Choosing a model on leaderboard position. Benchmark suites measure aggregate performance on public tasks, and a model that ranks third overall may be first on your domain. Skipping the evaluation set means the decision was made on someone else's data.

Filling the context window because it is available. Long prompts raise cost and latency and dilute attention across irrelevant material, and retrieval precision usually matters more than retrieval volume.

Shipping without transcript review. Generative failures are fluent, so nobody reports them the way they report a broken flow. Teams that discover problems from customer complaints are learning about them weeks late, a pattern discussed in deflection rate versus resolution rate.