Function calling in large language models (LLMs)

What is function calling in LLMs?

Function calling in LLMs is the mechanism by which a language model, shown a catalogue of typed function definitions, responds to a user turn by emitting a structured call: a function name and a JSON argument object that your own application code executes and hands back to the model.

The model never runs anything itself. It writes a request; your runtime validates the arguments, calls the API, and returns the result as a new message. Every function definition you expose occupies tokens on every turn, so a catalogue of fifty tools costs budget before the customer has typed a word.

How function calling works

Function calling runs as a five-step loop, and the model participates in only two of the steps.

First, the application sends the tool schemas alongside the conversation. Each schema names a function, describes when it should be used, and declares its parameters with types and permitted values. Those definitions sit in the context window next to the customer's message, which is why schema wording behaves like prompt engineering.

Second, the large language model decides. Given the turn and the catalogue, it either replies in text or emits a call, populating each argument from free-form customer phrasing that rarely matches your field names.

Third, your code validates those arguments against the schema and rejects anything malformed or unauthorized. Fourth, it executes the call against the real system. Fifth, the result is appended to the conversation and the model is invoked again, either to phrase an answer or to issue a further call. An AI agent framework is largely plumbing built around that loop, plus the rules for when to exit it.

Types of function calling

• Read-only lookups: Functions that fetch state such as order status or account balance and change nothing, so they are safe to retry and cheap to expose broadly.

• Write actions: Functions that mutate a record, cancel a subscription, or move money, where a wrong argument becomes a customer-visible incident within seconds.

• Parallel calls: The model emits several independent calls in a single turn, which cuts latency when the calls genuinely have no dependency between them.

• Sequential calls: Each call's output supplies the next call's arguments, so an early extraction error propagates silently down the chain.

• Forced calling: The application constrains the model to a specific function or to any function at all, useful when free choice produces unreliable routing on ambiguous turns.

Function calling vs tool calling vs structured outputs vs AI agent frameworks

Four names circle the same behaviour, and teams use them interchangeably until an integration review forces a distinction. Tool calling describes the same mechanism at a broader scope, covering retrievers, code interpreters, and remote services alongside functions. Structured outputs describe a model constrained to emit JSON matching a schema, with no execution implied. AI agent frameworks describe the software layer that runs the loop, keeps state, and decides when to stop. Function calling is the protocol underneath all three: one model request for execution, expressed as a validated argument object.

If you need the model to act once on one system with predictable arguments, function calling is the whole answer. If you need several turns, retries, memory between them, and a stopping rule, you need the framework layer wrapped around those calls.

Why function calling matters for customer experience

Without function calling, a support model is limited to describing your policies. It can explain the return window accurately and still leave the customer to log in, find the order, and start the return by hand, which is the point where most self-service journeys are abandoned.

The sharper failure is what a model does when asked about live state it cannot reach. Pressed for a delivery date with no function available, a fluent model will produce one, and the customer receives an AI hallucination formatted as a shipping update.

The tradeoff is direct: every function you expose widens both what the agent can resolve and what it can get wrong unsupervised. Read-only lookups earn their place quickly. Write actions on money or accounts have to justify themselves against the cost of one bad argument repeated at volume.

How is function calling measured?

Measurement splits into two halves, and only the first has public scoreboards.

Public function-calling evaluations cover the model's raw capability. Leaderboards such as the Berkeley Function Calling Leaderboard score models on whether they select the correct function and populate its arguments from a fixed catalogue of public APIs. That taxonomy is a general one: your schemas, your permitted values, your customer phrasing, and your authorization rules are absent from it, so those scores rank models without predicting how yours will behave on your catalogue. Nor does any standards body publish a target accuracy a support team is expected to hit.

The second half you measure yourself, on a labelled set of real transcripts. Track call-selection accuracy, argument accuracy, the rate of unnecessary calls, and the rate of refusing to call when action was required. Grade each metric per function, because one badly described schema drags the aggregate. The NIST AI Risk Management Framework describes the measure function that this kind of continuous, context-specific evaluation belongs to.

How AI agents change function calling

A single call answers a question. An agent turns the call into one step inside a plan: it calls a lookup, reads the result, chooses the next call from what came back, and stops when a goal condition is satisfied. That loop converts a model that can talk about a refund into one that can issue it, which is the substance of action-taking support automation.

The autonomy moves where the risk sits. In a single-call setup, a bad argument produces one wrong lookup that a human can catch. In a loop, that same bad argument becomes the input to the next call, and the model reasons confidently over the corrupted result all the way to a customer-facing answer.

Teams answer this with approval gates on write functions, per-function retry limits, and a hard cap on loop iterations, so an agent that has lost the thread halts before it spends the afternoon calling your API.

What to look for when implementing function calling

Coverage comes first: list the contact reasons you want resolved, then check whether each one has a system of record with an API that can be called synchronously. Reasons without an endpoint stay conversational no matter how good the model is.

Integration surface decides the build. Functions that wrap a documented REST API take days; functions that wrap a legacy screen-scraping path take quarters. Ownership matters as much, because a schema is production code and needs a named owner and a review trigger when the underlying endpoint changes.

Two frameworks genuinely bite here. SOC 2 Type II governs the credentials your function layer holds, since a support agent calling a billing API is a privileged service account. GDPR forces data minimisation on arguments and traces, because every logged call preserves whatever personal data the model extracted. The guardrails behind trustworthy automation live at that layer.

The constraint most teams meet late is idempotency: models retry, and a refund function without an idempotency key will issue two.

Function calling and agent orchestration

Function calling is the smallest unit that AI agent orchestration coordinates. The orchestrator decides which agent owns a request; the calls are how that agent changes anything in the world. When several agents share one tool catalogue, the orchestrator also owns collision rules, so two of them cannot both issue the same credit.

The other dependency is state. Arguments frequently come from earlier turns, so AI agent memory supplies the order number the customer gave three messages ago. A memory gap surfaces as a call with a null argument, which reads like a tooling bug and is really a context problem.

What does function calling mean in plain terms?

Think of the model as a waiter. It listens to the customer, writes a ticket in the kitchen's exact format, and hands it through the window; the waiter never cooks anything. LLM stands for large language model, and that model is the waiter: it turns "where's my order, I paid Tuesday" into a ticket reading get_order_status with a customer ID and a date range.

Take the ticket format away and the kitchen has to guess. The model would either decline to help or describe an order it never looked up, and the customer would believe the description because it sounds exactly like a real one.

The tradeoff is that a waiter with keys to the whole kitchen can plate the wrong dish. Every function you hand the model is a live capability with live consequences, and the schema is your only chance to say what it may touch.

Common function calling mistakes

Vague schema descriptions cause most routing errors. The model selects a function by reading its description, so two functions whose descriptions overlap produce something close to a coin flip, and the fix is editorial rather than technical.

Treating model output as trusted input causes the expensive ones. Arguments are generated text that happens to look like data. Skipping schema validation and a server-side authorization check lets an invented account ID reach a production database with the agent's credentials.

Exposing the entire API is the third pattern. Mapping every endpoint to a function inflates the catalogue, consumes context on every turn, and raises mis-selection, because the model now chooses among a dozen near-identical lookups.

Ignoring idempotency is the quiet one. Timeouts trigger retries, retries reissue calls, and a write function with no deduplication key turns one refund into two before anybody reads the trace.