What is latency?
Latency is the elapsed time between a customer's input and the system's response, measured in milliseconds or seconds. In support it is tracked end to end, from the moment a message is sent or a caller stops speaking to the moment a usable reply arrives.
Thresholds are set by human conversation. In voice, a reply under roughly 800 milliseconds feels like a normal exchange, and past about 1.5 seconds callers begin talking over the agent or hanging up. Text channels tolerate a little more before the same impatience appears.
How latency accumulates across a support stack
End-to-end latency is additive. Every hop between the customer and the answer contributes, and the figure the customer feels is the total of all of them.
Five layers carry most of the budget. Transport is network time in both directions. Input processing converts what arrived into something a model can read: transcription for voice, parsing and normalisation for text. Retrieval fetches the policy or account record the answer depends on, and a slow search index surfaces here first. Inference is generation time, which scales with output length and model size, so an LLM router that sends simple queries to a smaller model trims it directly. Action execution is the fifth: every tool call to a CRM or order system adds that system's own response time to yours.
Two properties make the sum hard to control. Hops run sequentially by default, so each one waits for the last to finish. And any hop close to saturation adds queueing time on top of its own work. Precomputing what can be precomputed, which is what batch inference does for embeddings and summaries, removes that work from the live path entirely.
Types of latency in a support stack
Time to first token: How long before the first fragment of a response appears, which is what makes a reply feel like it has started.
Time to full response: Total time until the complete answer exists, the figure that governs any channel where nothing streams.
Turn latency: The silence between a caller finishing and the agent beginning, the gap that speech-to-speech architectures exist to compress.
Action latency: Time to complete the work behind the answer, such as issuing a refund, which frequently exceeds generation time.
Perceived latency: What the customer experiences, which streaming, typing indicators and filler audio shorten without changing the underlying measurement.
Latency vs first response time vs throughput vs jitter
Speed metrics get used interchangeably in dashboards, and the substitution hides which problem you actually have. First response time measures how long a ticket waits for its first reply, in minutes or hours. Throughput measures how many requests the system serves per second under load. Jitter measures how much consecutive response times vary around their own average. Latency measures the wait a single customer sits through on one turn, which is the only one of these four a person experiences directly.
What it counts | What it misses | Typical benchmark | |
|---|---|---|---|
Latency | Elapsed time from input to response, per hop and end to end | Whether the answer was correct or complete | Under roughly 800 ms feels conversational in voice |
First response time | Minutes or hours until a ticket receives its first reply | Every delay inside the conversation after that reply | Set by channel SLA; no cross-industry norm |
Throughput | Requests served per second at a given capacity | What any individual customer waited through | A capacity-planning figure, specific to your stack |
Jitter | Variation between consecutive response times | The absolute speed of any single turn | Judged by the spread between median and p99 |
Two of the four move together: pushing throughput up against fixed capacity raises per-request latency, because the extra requests queue before they are served. If customers complain about waiting, measure latency at p95. If the system buckles at peak while single requests look fine, measure throughput. If replies are quick on average yet feel erratic, jitter is the number to chase.
Why latency matters for customer experience
Latency is the one quality signal a customer can judge without knowing anything about your architecture. Silence reads as failure long before it reads as thinking.
When latency goes unmanaged, the damage arrives in a predictable order. Callers talk over the agent because the pause read as a dropped line, then they repeat themselves, which corrupts the transcript, then they hang up, which is where call abandon rate starts moving. In chat the same delay produces double-sent messages and duplicate tickets, so a latency problem arrives at the team as a volume problem.
The tradeoff is genuine and unavoidable. Almost every technique that improves answer quality costs time: retrieving more documents, calling a second tool to verify an account, running a guardrail check on the draft. Each adds hundreds of milliseconds and each makes the answer more likely to be right. Teams that optimise for speed alone ship fast wrong answers; teams that ignore speed ship correct answers nobody stayed for.
How is latency measured?
Latency is a distribution, and it should be reported as one. Record a timestamp at every hop boundary, then publish the median, the ninety-fifth percentile and the ninety-ninth. The median flatters a demo. The upper percentiles describe the customers who are genuinely annoyed, and those are the people who abandon, write reviews and open a second ticket about the first one.
Instrument per hop as well as end to end, because a bad total tells you nothing about which layer owns it. Measure under peak traffic, since queueing at a saturated dependency is invisible when a single request is in flight.
No cross-industry body publishes a target for AI response latency, so the honest external anchor is what the wait costs. The U.S. Bureau of Labor Statistics reports median pay for customer service representatives at USD 20.59 per hour, or USD 42,830 per year, which is the labour cost that every abandoned automated turn pushes back into a human queue.
How AI agents change latency
Deterministic bots had predictable timing because they ran a fixed script. AI agents made latency variable at runtime. Generation time depends on output length, retrieval depth and how many tools the agent decides to call, and the agent decides those mid-turn, so two questions about the same order can differ by several seconds.
That variability changed the engineering response. Streaming partial output covers generation time. Speculative retrieval starts fetching before the intent is fully resolved. Parallel tool calls collapse three sequential dependency waits into the length of the slowest one. None of these makes the work smaller; they change where the customer sits while it happens.
The deeper consequence is that latency became a constraint on how autonomous an agent is allowed to be. Every verification step buys accuracy with time, and the budget is tightest in voice, where the standards for low-latency voice agents leave almost no room for a slow lookup.
How to reduce latency in production
Reduction starts by giving each hop an explicit allowance and finding which one is spending beyond it. Four axes decide how much room you have.
Coverage: decide which intents and channels the budget applies to, since a refund that touches a payment processor cannot hold a voice-grade target. Integration surface: dependencies you do not own set your floor, so cache what is safe to cache and parallelise what has no ordering requirement. Governance: one named owner holds the p95 figure, because a number reported by everyone is improved by nobody. Compliance: regulated buyers ask where inference happens, and answering GDPR data-residency questions often pins processing to a single region, which sets a transport floor no amount of tuning removes.
The operational constraint teams underestimate is cold starts. Infrastructure scaled to zero overnight will greet the first caller of the morning with the worst latency of the day, a pattern covered in these customer response time techniques.
Latency and conversational channels
Latency behaves differently on every surface a conversation runs on. In live chat a typing indicator buys several seconds of tolerance, because the customer can see that work is happening and can look away without losing the thread.
Voice removes that cover entirely, which is why any conversational AI deployment that spans both channels needs two separate latency budgets rather than one shared target. Email inverts the problem: minutes are acceptable, so the constraint moves to peak-hour queue depth.
What does latency mean in plain terms?
Think of latency as the pause before someone answers you at a service counter. If the clerk looks up, says nothing for four seconds, then gives a flawless answer, you have already decided they were unsure of it. The content of the answer never recovers the ground the pause lost.
Remove the pause and the identical sentence reads as competence. That is the whole of what latency work buys: the same answer, believed.
The tradeoff sits on the other side. The fastest possible reply is the one that skipped checking your account, and it will occasionally be confidently wrong about your balance. Every hundred milliseconds you cut has to come from somewhere, and the honest question is always which check you are willing to stop running.
Common latency mistakes
Reporting the mean is the first. An average absorbs the tail, so a stack where one request in twenty takes eight seconds can post a comfortable number while producing a steady stream of abandoned conversations that nobody can trace to timing.
Optimising the model while a dependency owns the delay is the second. Teams spend a quarter shaving inference time when the actual cost sits in a CRM lookup, and the end-to-end figure barely moves because the bottleneck was never touched.
Testing at idle is the third. Queueing only appears near saturation, so a stack that is quick in staging can degrade badly under production spikes, and the degradation lands hardest on the busiest hour of the week.
Treating perceived latency as the whole fix is the fourth. Streaming and filler audio genuinely help, but a reassuring phrase followed by six seconds of silence while a refund posts costs more trust than the raw wait did.
What is a good latency for an AI support agent?
Good latency depends entirely on the channel. Voice needs a turn under roughly 800 milliseconds to feel conversational, and past about 1.5 seconds callers interrupt or hang up. Chat tolerates a few seconds when a typing indicator is visible. Email measures in minutes. Judge every target at the ninety-fifth percentile.
What is the difference between latency and response time?
Latency and response time describe overlapping windows at different scales. Latency measures one turn: the gap between a customer's input and the system's reply, in milliseconds or seconds. First response time measures how long a whole ticket waits before anyone or anything answers it, usually in minutes or hours. One is felt live; one is reported weekly.
Latency vs throughput: which one should support teams track?
Latency and throughput answer different questions and both belong on the dashboard. Latency tells you what an individual customer waited through on a single turn. Throughput tells you how many requests the system can serve per second before it degrades. Under fixed capacity they trade against each other, since extra load makes requests queue.
Why does p95 latency matter more than average latency?
P95 latency describes the slow tail that averages hide. If nineteen requests return in half a second and the twentieth takes eight seconds, the mean still looks healthy while one customer in twenty experiences a broken conversation. Those are the people who abandon, repeat themselves, or open a duplicate ticket about the delay.
What causes high latency in AI customer support?
High latency in AI customer support accumulates across hops: network transport, transcription or parsing, retrieval from a knowledge source, model generation, and every external tool call to a CRM or billing system. Sequential execution compounds it, and any dependency running near capacity adds queueing time on top of its own processing.
How does latency affect voice AI compared with chat?
Latency punishes voice far harder than chat. In voice, silence has no visual cover, so a pause past about 1.5 seconds reads as a dropped call and triggers interruption. In chat, a typing indicator makes several seconds acceptable because the customer can see work is underway. Cross-channel deployments need two separate budgets.