The accuracy number in the demo is not the number you get in production

There are two accuracy numbers, and they are not the same number

When a voice agent demos at “99% accuracy,” that almost always means word error rate measured on a clean benchmark: a cooperative speaker, a good microphone, a quiet room. Top streaming speech-to-text models genuinely do score in the 2 to 4% word error rate range on those benchmarks. That part is real, and it is the easy part.

The number that decides whether your agent is usable is the other one: did it complete the task. For a phone-ordering agent, did the right ticket fire, priced correctly, with the right modifiers. For a booking agent, did the right reservation land. That is task completion, and it is a different measurement from transcription. You can transcribe every word perfectly and still complete the wrong task, because the words that carry the task (a quantity, a name, a modifier, a menu item) are a tiny fraction of the words spoken.

Getting “large” wrong instead of “small” is one wrong word in twenty. That reads as 95% transcription accuracy. It is also a 100% wrong order. This is why teams that take production seriously track a separate slot error rate on the words that actually carry meaning, and why a single headline accuracy figure tells you almost nothing about whether the agent works.

How far the demo number falls on a real line

The degradation is measured and consistent. A contact-center analysis ran the same speech API across real call conditions and found that acoustic conditions, not the model, explained most of the variance. Here is what the same agent looks like when you stop feeding it clean audio.

Figures from Deepgram’s production accuracy analysis: clean headset 92%, conference room 78%, mobile call with background noise 65%, accented speech without tuning below 80%, and a documented 2.8 to 5.7 times benchmark-to-production degradation. A restaurant phone line, on a narrowband codec with kitchen noise behind the caller, sits at the worse end of that range.

The demo is engineered to be the best case

Signal-to-noise ratio alone drives an exponential collapse: roughly 3.5% word error rate at 20 dB SNR, around 15% at 10 dB, and over 70% at 0 dB. Microphone bandwidth compounds it. A narrowband phone channel (300 Hz to 3.4 kHz) can sit near 25% word error rate at 10 dB where a wideband channel would be around 12%. A demo runs at the top of every one of those curves. A Tuesday-night phone call runs near the bottom.

Accents are the other quiet failure. Most models are trained mostly on standard American English, so accuracy that reads 96% on benchmark audio can fall below 80% on heavily accented speech. Targeted acoustic training closes some of it (one study moved accent handling from 76% to 88% with 200+ hours of data), but the default out-of-the-box number you saw in the demo did not include any of your callers.

Demo conditions vs production conditions

The single most useful thing you can do before trusting a voice agent is to stop testing it the way it was demoed. Toggle between the two worlds it lives in.

The metric that survives production: did the right thing happen

Once you accept that raw transcription will degrade on a real line, the design question changes. It is no longer “how do we hear every word,” it is “how do we still complete the task when we do not.” For a restaurant phone agent, that means the output is not a transcript, it is a structured order that has to resolve to a real menu item, a real modifier code, and a real price. A correct transcript with no menu behind it is still a ticket nobody can ring up. We broke that specific failure down separately in why transcription accuracy is not POS accuracy, and the “ask 95% of what” problem in our breakdown of restaurant phone order accuracy.

Because the line itself is the worst case for raw transcription (narrowband codec, kitchen noise, a phone held to the ear), the agent has to lean on structure and confirmation rather than perfect hearing. The acoustic side of that problem, and what actually moves the number on a noisy line, is its own topic in our note on the acoustic noise bottleneck.

How PieLine is built for the gap, not the demo

Two design choices follow directly from everything above. They are the part you can hold a vendor to.

What to ask before you trust the accuracy number

Skip the headline figure. It is the cheapest thing to quote and the hardest thing to verify. Ask the questions that expose whether the number was ever measured under production conditions:

• Is that accuracy word error rate or task completion? If it is word error rate, ask what it drops to on real calls.
• What audio was it measured on? Clean headset audio and a narrowband phone line on a noisy floor are two different planets.
• What happens to the calls it cannot complete? A safe handoff with context beats a confident wrong answer every time.
• Can I see the number on a sample of my own calls? The only accuracy figure that matters is the one measured on your callers, your noise, and your menu.