Voice activity detection (VAD)

What is voice activity detection?

Voice activity detection (VAD) is the signal-processing step that decides whether a slice of audio contains human speech or only silence, noise, and line hiss. It runs continuously on a live stream and emits a speech or non-speech label many times per second, and everything downstream in a voice pipeline waits on that label.

Timing is the reason it matters. Human turn-taking gaps average around 200 milliseconds, and a pause past half a second starts to feel like the other party has stopped listening, so a detector that waits too long makes an otherwise capable voice agent feel slow.

How voice activity detection works

Classic VAD splits the incoming stream into short frames of 10 to 30 milliseconds, which gives roughly 33 to 100 decisions per second, then computes cheap features on each one: short-term signal energy, zero-crossing rate, and spectral flatness. A threshold turns those features into a binary label, and a hangover timer holds the label at "speech" for a few extra frames so a natural pause inside a sentence survives as part of the same turn.

That label gates the rest of the stack. Automatic speech recognition receives only the frames marked as speech, which cuts compute and keeps phantom words out of the transcript. Barge-in depends on the same signal in reverse: when the detector fires while the agent is talking, playback of the speech synthesis output stops within a few frames and the caller is heard mid-prompt. Endpointing sits on top of all of it, converting a run of non-speech frames into a decision that the caller has finished and the agent may reply.

Types of voice activity detection

• Energy-threshold VAD: Compares frame energy against an adaptive noise floor, cheap enough to run on a handset DSP, though it happily labels a slamming door as speech.

• Statistical model VAD: Scores each frame against learned distributions for speech and for noise, which holds up on steady background hum but weakens against babble.

• Neural VAD: A small trained network labels frames from spectral features and survives noisy telephony where fixed thresholds collapse, at the cost of a little buffering latency.

• Semantic turn detection: A language model judges whether the words so far form a finished thought, catching the trailing pauses that acoustic methods clip.

• Device-side gating: Always-on low-power detection that opens the network stream only once speech is present, useful for battery and privacy, blind to anything under its threshold.

Voice activity detection vs endpointing vs noise suppression vs speaker diarization

These four run on the same audio in the same second, which is why teams spend a week debugging the wrong one. Endpointing decides when a caller's turn has ended and the agent may speak. Noise suppression removes non-speech energy from the signal so what remains is cleaner audio. Speaker diarization attributes each stretch of speech to a particular talker across the call. Voice activity detection answers the narrower question underneath all three: is anyone speaking in this frame at all.

If your agent talks over callers or leaves dead air, start with the detector and its hold time. If the transcript is full of phantom words, treat it as a suppression and threshold problem. Diarization only pays off once you are analysing recordings.

Why voice activity detection matters for customer experience

When detection is tuned badly the symptoms reach the caller directly. A hold time that is too long produces dead air, and callers fill it by repeating themselves, which arrives as a second overlapping utterance the agent now has to reconcile. A hold time that is too short cuts people off mid-sentence, most often the callers who pause to read an account number off a card. An AI IVR that replaced keypad menus with natural speech inherits this as its central timing problem, because there is no keypress marking the end of an input.

The tradeoff is fixed. A 700 millisecond hold, 300 milliseconds of model time, and 400 milliseconds to start audio is a 1.4 second gap; cutting the hold to 400 milliseconds brings it to 1.1 seconds and removes the same 300 milliseconds of margin for slow talkers.

How is voice activity detection measured?

Detection quality is scored on two errors that trade against each other: missed speech, where a frame of real talk is labelled silence, and false alarm, where noise is labelled speech. Evaluations combine them into a single detection cost, weighted by how expensive each error is in the target application.

Published numbers are scarce because most detection is embedded inside a product. The nearest standing public reference is the NIST Speaker Recognition Evaluation, which scores speech activity detection as a component task and reports miss and false-alarm rates across a range of audio conditions, from clean interview microphones to degraded telephone channels.

In production the number teams actually tune is endpoint latency: the gap between the caller falling silent and the agent starting to speak. Measure it on recorded calls, segmented by line quality, since a hold time that feels crisp on a quiet office line clips a caller on speakerphone in a car.

How AI agents change voice activity detection

The mechanism change is that the decision stopped being purely acoustic. An LLM-driven agent already holds a partial transcript, so it can ask whether the words so far form a complete thought and keep the floor through a pause that an energy threshold would have ended. Some speech-to-speech architectures go further and fold turn handling into the model itself, so no separate detector emits labels for a downstream component to trust.

The consequence is that turn timing became a conversation design decision owned by the people writing the flows, and it left the DSP config file. Teams evaluating voice agent platforms for support now test barge-in behaviour and hold time on recordings pulled from their own queue, because tuning that works for a two-word checkout question fails on a caller reading a serial number aloud.

What to look for in a voice activity detection stack

Judge it on four axes. Noise robustness comes first, and it has to be tested on the codec you actually carry: narrowband telephony discards everything above roughly 3.4 kHz, so a detector trained on wideband recordings loses cues it relied on. Integration surface is second: the detector has to sit where it can act, in the browser, on a SIP media server, or inside the platform pipeline.

Ownership is the axis teams skip. Somebody has to own hold time per queue and retune it when the call mix shifts. On governance, health and finance buyers ask where the pre-roll audio buffer lives and how long raw audio is kept before recognition, and HIPAA and GDPR reviews turn on that answer, so document it before the review starts.

The constraint that bites in production is packet loss. A jitter buffer starved of frames looks exactly like a caller who stopped talking, so a detector with no transport awareness will end the turn on a network problem.

Voice activity detection and voice authenticity

Everything that interprets a call downstream depends on clean segments arriving first. Emotion detection reads prosody across a stretch of speech, so segments that start late carry clipped pitch contours and produce readings nobody acts on. Speaker verification has the same dependency plus a newer problem: voice cloning means a segment correctly labelled as speech may still be synthetic, so systems that gate account actions on a voice now run a liveness check as a separate step after detection.

What does voice activity detection mean in plain terms?

VAD stands for voice activity detection, and the full form describes the job precisely: it detects when voice activity is present. Think of it as a doorman for the microphone, deciding which sounds walk through to the rest of the system and which are left outside as noise.

With no doorman, everything reaches the transcriber: the air conditioning, a dog, the colleague two desks over, and the agent answers a question the caller never asked. With one that is too strict, a soft-spoken caller knocks and is left on the step.

The tradeoff shows up in every deployment. Making the doorman quicker to open the door means fewer people wait, and it also means more of the street gets in.

Common voice activity detection mistakes

Tuning on clean audio. Thresholds calibrated on studio-quality recordings collapse on a hands-free call in traffic, because the noise floor they were fitted to no longer exists. Tune on a sample drawn from the live queue, including its worst lines.

Treating silence as the only end-of-turn cue. Callers pause to think, to read, and to check with someone else in the room. An acoustic detector reads all three identically, which is how hold times creep upward until the agent feels sluggish to everybody.

Fixing interruptions by desensitising the detector. When an agent keeps cutting itself off on echo, the tempting fix is a higher threshold, and the price is every quiet caller. Echo cancellation is the layer that should carry that repair.

Shipping with no fallback. Detection will fail on some calls, and the design question is what happens then: a repeated prompt, a DTMF option, or a human fallback path. Calls with repeated detection failures are the ones worth routing to a person early.