Evaluating speech-to-text models

9 July 2026 Jess Wang 21 min

If you're building a voice agent, picking the right speech-to-text model is not obvious. Every provider claims to be accurate, fast, and production-ready, and the benchmarks they publish rarely look anything like your actual traffic. The only way to know is to run your own eval on audio that resembles what your users will say.

I ran a controlled eval in Braintrust across six STT providers, 240 audio cases, and eight content domains, scoring not only whether each model heard the words but whether its mistakes changed the downstream answer. This blog walks through how to set up a voice eval, what to measure, the tradeoffs that decide the winner, and which models came out on top.

What to measure

In a common voice agent pipeline, audio comes in -> an STT model turns it into text -> an LLM reasons over that text -> a reply goes out. In this pipeline, two things matter:

1. Did STT hear the words right?
2. If there is a transcription error, how much does it affect the final response?

To measure the second question, I ran the same LLM prompt twice per case. Once on the reference transcript (the known-correct text, which acts as my answer key) and once on the STT output. Then I checked whether the two answers mean the same thing using an LLM-as-a-judge scoring system.

The dataset

I collected 240 audio files total to test against each STT model. Those audio files were spread across eight content buckets, with 30 cases each.

The seven Hugging Face buckets were pulled through the dataset viewer REST API instead of the datasets library, which sidesteps torchcodec and the gated-dataset walls. It paginates 100 rows per request, and ffmpeg normalizes every clip to 16 kHz mono before scoring.

The customer_support clips were read in British, American, and Australian accents. I used ElevenLabs text-to-speech to create the audio clips.

Here are the six models I compared:

Defining tokens that matter

Every case carries a critical_entities list. These are the specific tokens that are more important to transcribe correctly for the final answer to be relevant. For example, dropping a comma isn't too bad. But if you drop the last digit of an order number, the whole response breaks.

For customer_support I wrote these critical entities by hand because I know which IDs and actions each script contains:

• A100, cust_123, INV-4492, SN-90455-K (order numbers, customer IDs, invoice and serial codes)
• charged twice, cancel, subscription, refund (the actions that decide what the agent does)

For the seven remaining buckets there's no human curating those lists, so I auto-extract them with a small heuristic. It scores every token by how "hard" it is and keeps the top four.

The scoring system

I had four scorers in place:

1. transcription_similarity_score is lexical (word-level) similarity between the reference and the STT output. Edit-distance ratio over normalized tokens.
2. critical_entity_recall_score is how many of the hard tokens survived.
3. answer_equivalence_score is the LLM-as-a-judge on whether the answer from the STT transcript means the same as the answer from the reference answer key.
4. stt_latency_ms is per-call STT speed.

Do the scorers measure the right thing?

Two of the three scorers are deterministic. The one worth checking is the LLM-as-judge (answer_equivalence). If it's sound, its verdicts should line up with the deterministic scorers. Pooling all 1,439 baseline cases, that is what happens.

The ordering is strictly monotonic, and the full breakdown is a clean diagonal. High-similarity cases concentrate in "same" and low-similarity cases in "different."

The model comparison

Each model ran as its own Braintrust experiment against the identical 240 cases. Here are the results:

The accuracy columns are a near-tie, and it is a real tie, not merely close averages.

The structured-token problem

Here's critical-entity recall broken out by bucket and model.

Philosophy contains a lot of dense old-book vocabulary, words like PARAPHERNALIA, EXTINGUISHED, and MISFORTUNE, and every model sits at ~0.97 for that bucket.

Fix 1: glossary biasing

In an attempt to boost these scores, I tried handing the model some missing context.

Every major STT API takes some kind of vocabulary hint. I built one glossary per bucket from external, standard vocabulary and fed it into each provider's native slot.

Two rules kept it honest. First, every glossary is built from external, standard domain vocabulary, never from the clips' own reference transcripts. Second, for customer_support that means made-up example IDs.

Fix 2: post-transcription correction

There's a fix that works for all of them. Run an LLM over the finished transcript and let it apply the format rule as a proofreading pass.

On customer-support entity recall it's the best of the three approaches for every model, and it rescues the exact providers biasing couldn't help.

Main takeaways

1. Structured tokens are the failure mode. Ordinary words, even fancy ones, transcribe fine.
2. Accent variation barely moved the scores.
3. The major models are basically tied on accuracy.
4. Speed is the real differentiator.
5. The cheap wins are biasing and post-correction.
6. What's different for voice is that you have to listen.

No single model wins everything, so match the model to what you care about. If your STT provider accepts a prose prompt, hand it your formats. If it doesn't, add an LLM post-correction pass.