The models scoring best on the VoxPopuli test set are the ones that most often copy its faulty reference transcript rather than transcribe the recording.
Hugging Face tested 11 widely used open-source speech recognition models and found that several of the best scorers write out the transcript filed with the benchmark instead of the words in the recording. For those models, the published scores overstate how well they transcribe speech in general. The clips came from VoxPopuli English and from LibriSpeech in both its clean and other splits.
The pattern is widely discussed in machine learning, under labels such as benchmark optimization or "benchmaxxing," but speech recognition has resisted attempts to measure it.
One VoxPopuli clip can be heard to contain "Thank you, Mr. President," while the reference transcript supplied with the benchmark leaves out "Thank you." Six of the 11 models left it out as well, giving the answer the benchmark expected against the audible evidence, and those same six copied the benchmark's punctuation, writing "Mr" with no period; models that caught the phrase generally wrote "Mr." with one.
The researchers then had the same line spoken by a cloned voice, built from a parliamentary recording made after every model's training data stopped. Every model but one transcribed what it heard. Re-rendered in a generic text-to-speech voice with no tie to any parliamentary recording, the line drew the courtesy phrase from all eleven.
They also gathered fresh post-cutoff material from the same kinds of sources — recent European Parliament recordings, readings by LibriVox narrators only lately active — and on that audio many models stopped matching the reference and moved back toward the sound. On this evidence the models hear the words accurately, then read the sound around them to decide whether to write what they heard or what one benchmark's transcription policy expects.
Three probes put numbers on how often this happens. A consensus disagreement probe asks whether a model writes what it hears or falls back on a faulty reference where the two conflict; among the models that showed the pattern, the faulty reference was copied in 18–30% of cases.
A masked entity retrieval probe mutes numbers in the recording, so no figure should come out of the model at all; on LibriSpeech, several of the top benchmark performers wrote the muted number anyway in roughly 30–40% of examples.
An orthographic switching probe uses forms that sound alike but are written differently: 1 against one, Mr. against mister, Honor against Honour. Across datasets several models beat the 50% that guessing would return, a few coming close to 90% switch accuracy — evidence they can tell which dataset a clip came from and supply the spelling that benchmark wants.
The same method flagged possible reference mistakes in 40% of the VoxPopuli test clips examined, touching about 3% of reference words overall. VoxPopuli is known to carry many transcription errors, and Artificial Analysis released a cleaned version of the dataset in response.
The Open ASR Leaderboard has gained a "Benchmark fitting" tab reporting each model's VoxPopuli reference error rate and its orthographic switching across the public datasets. The researchers argue for evaluation sets kept entirely unseen, and for judging a model on more than its word error rate on one public benchmark.