Your Dictation Is Less Accurate for Some Voices — Measure Your Own Error Rate Before You Trust a Voice Note

No speech recognition system has one accuracy. Peer-reviewed studies — including a 2020 PNAS paper that tested five commercial systems across 115 speakers — measured word error rates that differ substantially by speaker and by speech style. The published headline number describes a model's average. It does not describe what happens to your voice.¹

That distinction matters every time a dictated note becomes the record — a meeting summary, a doctor's note, a legal transcript, a quick thought captured while driving. If your transcription tool makes 35 errors per 100 words while quoting a different accuracy figure, you are not getting what you think you are getting. The difference between an accurate note and a rewritten one is invisible if you never check.

This piece reports what the studies measured, where current models like Whisper sit, the two distinct ways a transcript can go wrong, and a ten-minute procedure you can run right now to produce your own error profile.

Five systems, one matched corpus, one gap

No speech recognition system has a single accuracy. Koenecke et al. tested five commercial systems — developed by Amazon, Apple, Google, IBM, and Microsoft — on 19.8 hours of matched interviews with 42 white speakers and 73 Black speakers across five US cities, controlling for age and gender.¹ The average word error rate was 0.35 for Black speakers and 0.19 for white speakers — a near-doubling. The gap held at every vendor.

Word error rate (WER) is defined as the count of substitutions, deletions, and insertions divided by the total words in the reference.¹ A WER of 0.35 means that 35 of every 100 words were wrong in some way. The authors set a practical threshold: a WER above 0.50 renders a transcript unusable. At that threshold, 23% of audio snippets from Black speakers produced unusable transcripts, versus 1.6% from white speakers.¹

The variance across vendors was wide. Microsoft, which had the best overall performance in those tests, returned a WER of 0.27 for Black speakers and 0.15 for white speakers. Apple, with the weakest overall performance, returned 0.45 and 0.23, respectively.¹ In every case, error rates for Black speakers were nearly twice as large.

The researchers traced the gap to the acoustic models themselves — not to vocabulary or topic. They tested a subset of 206 matched utterances, phrases of five to eight words spoken by both speaker groups, and found the disparity was equally large on identical phrases. The mechanism was pronunciation and prosody: "rhythm, pitch, syllable accenting, vowel duration, and lenition."¹ The model was not confused by the content. It was less trained on how certain speakers sound.

These are 2019-era systems. The authors named them because they were the five largest commercial providers at the time. Do not treat those numbers as today's accuracy for any of those products — vendors have retrained repeatedly since. What the study established is the existence and mechanism of the gap, not its current size.

Not one study — and not every system shows it

The Koenecke finding does not stand alone. Tatman (2017) tested YouTube's automatic captions on speakers across eight English dialects and found lower accuracy for women and for speakers from Scotland, concluding the research "demonstrates the need for sociolinguistically-stratified validation of systems."² Liu et al. (2021) analyzed a publicly released 846-hour corpus and found "significant differences in word error rate across gender and skin tone… at times for all models" — the hedge "at times" is in the paper and should stay in any citation of it.³

But a concurrent 2017 study by Tatman and Kasten found something important enough to sit here as a direct counter: Bing Speech showed dialect and ethnicity WER differences that were not statistically reliable, while YouTube's captions did. Neither system showed a reliably different WER by gender — a variable that had been reported for YouTube's captions in earlier work.⁴

The upshot is not that one side is right. It is that the gap is system-specific and variable-specific. One tool in one year may show a substantial accent gap; a different tool may not; the same tool may show a gender gap in one study and none in another. The practical consequence is the same regardless of which side of a disparity you fall on: the headline accuracy figure does not tell you what the tool does on your voice. You have to measure.

Current models: the same direction, different framing

Whisper, OpenAI's open-weights speech recognition model, was evaluated in its own paper against two very different corpora: LibriSpeech, a clean read-aloud audiobook dataset, and the conversational interview corpus from Koenecke et al.⁵ Whisper Large V2 scored a WER of 2.7 on clean read speech and 16.2 on conversational interviews from that corpus — a six-fold difference, from the same model, with no change in settings.

That is a speech-style finding, not a fixed demographic number. Conversational interviews — with natural pauses, overlaps, and informal pronunciation — are harder for a model trained primarily on read speech and subtitles. The gap between 2.7 and 16.2 describes what happens when the training distribution and the real-world input diverge.

Graham and Roll (2024) evaluated Whisper across a range of accents and speaker traits and found superior recognition for American English compared to British and Australian accents, and higher accuracy for native than non-native English speakers. They also confirmed that Whisper "exhibits enhanced performance in read speech over conversational speech."⁶

One practitioner's benchmark — Nadine's 2026 measurement of Whisper on Nigerian English and Pidgin across 500 samples from five speaker backgrounds — found Whisper Large V3 at a WER of 35.05% after proper normalisation.⁷ Liu et al. (2021) independently found similar accuracy differences "at times" across skin tone and gender in an 846-hour corpus.³

The direction across these studies is consistent: conversational speech, regional accents, and non-native speakers tend to see higher WER in current models. The size of the gap varies — and the only way to know what it is for your voice and your tool is to measure it.

Two kinds of error: mishearing and rewriting

A transcript can fail in two distinct ways, and the second is harder to detect. The first is mishearing: the model fails to recognize a sound and outputs the wrong word or drops it. The second is rewriting: the model hears the word correctly but substitutes the standard-spelling form for the speaker's actual pronunciation.

Koenecke et al. found that the gap persisted even on identical five-to-eight-word phrases — the model was correcting away from the speaker's phonology, not being confused by new vocabulary.¹ Nadine's benchmark found that overcorrection accounted for 23.5% of non-faithful errors in Whisper Large V3 on her Nigerian English and Pidgin corpus. Her example is direct: "If a model transcribes 'pipo' (what the speaker said) but the reference says 'people', standard WER counts a substitution error. That penalizes a spelling difference, not a listening mistake."⁷

This rewriting pattern has real consequences. A participant in Mengesha et al.'s two-week diary study of 30 African American users of voice technology put it plainly: "It needs to change because it doesn't feel inclusive when I have to change how I speak and who I am, just to talk to technology."⁸ That study found that 23% of participants (7 of 30) reported voice technology failures when sending, replying to, or dictating a message.

WER penalises both error types identically — a limitation Radford et al. acknowledge directly: "WER penalizes all differences between the model's output and the reference transcript including innocuous differences in transcript style. Systems that output transcripts that humans consider correct can still have a large WER due to minor formatting differences."⁵ Their own normalizer carries a self-described risk: "there is a risk that our normalizer is overfitted to fixing Whisper's peculiarities rather than addressing general variation in transcription."⁵

The implication: when you measure your own WER, apply the same simple normalisation to both reference and transcript — lowercase everything, strip punctuation — and do not tune it to flatter the tool. The substitution list you get afterward is more diagnostic than the number itself.

The ten-minute check: measure your own word error rate

Measuring your own word error rate takes about ten minutes. Read a passage you know verbatim, transcribe it with your usual tool, normalise both texts the same way, then compute WER. The resulting list of substitutions is your personal error profile — the words your tool reliably changes, which you can search in older notes to find where corrections silently replaced what you said.

Here is the procedure:

Step 1 — Choose a known passage. Pick 150–200 words you know verbatim: a paragraph from a book you have at hand, a set of sentences you composed yourself. The requirement is that you have an exact reference text.

Step 2 — Dictate it naturally. Speak at your normal pace. Do not slow down or over-enunciate. The goal is your real dictation pattern, not the most favorable one.

Step 3 — Normalise both texts identically. Convert both the reference and the transcript to lowercase. Strip all punctuation. The jiwer Python package (v4.0.0 on PyPI) includes ToLowerCase and RemovePunctuation transforms.⁹ Apply them to both files — never only one. Radford et al. found WER can drop by up to 50 percentage points from a single formatting artifact in the reference transcript.⁵ Normalising both texts eliminates that distortion.

Step 4 — Compute WER and read the substitution list. jiwer.process_words(reference, hypothesis) returns a .wer float and an alignment. jiwer.visualize_alignment() prints each substitution pair in readable form. The diagnostic value is in the list, not the number.

Step 5 — Search your older notes. Take the substitution list and search your past voice notes for each transformed word. The ones your tool reliably rewrites are the notes most likely to contain the model's word, not yours.

If your note format is plain text, this search is a simple grep. The post on voice notes and local storage covers how format affects what you can do with a transcript after capture.¹⁰

What the studies do not prove

These findings are averages over matched corpora, not predictions for individuals. Koenecke's systems were measured in 2019. None of those vendor-level numbers apply to any current product version. The Tatman and Kasten study found that Bing Speech's dialect and ethnicity differences were not statistically reliable at all.⁴

The finding that per-speaker error rates differ from population averages is robust across multiple independent studies and multiple model generations. The specific size of the gap for any particular system today is not established by any of these papers.

WER is also an imperfect metric. It counts every deviation — including punctuation style, filler word handling, and contraction spacing — as an error. Radford et al. found WER drops of up to 50 percentage points from a single formatting quirk in a reference transcript — a formatting artifact, not a listening failure.⁵

This is the argument for measuring your own voice rather than relying on benchmark numbers. A benchmark tells you about the average speaker in a specific corpus. Your measurement tells you about you: which substitutions happen, how often, and on which words.

Koenecke et al. concluded by proposing "using more diverse training datasets that include African American Vernacular English" to reduce performance differences and "ensure speech recognition technology is inclusive."¹ That is a research roadmap, not a current product claim. Until models are publicly evaluated on the full range of accents and speech styles they are deployed on, the personal measurement is the most reliable data point available.

Frequently Asked Questions

Why is my real-world speech recognition less accurate than the advertised number?

Benchmark accuracy figures are measured on specific corpora — often read-aloud audiobooks or scripted speech. Koenecke et al. found that five commercial systems returned average WER of 0.35 on conversational interviews with Black speakers, against lower published figures. Radford et al. found Whisper Large V2 at 2.7 WER on clean audiobooks and 16.2 WER on conversational interviews. The advertised number and the deployed number describe different speech styles.¹⁵

Does Whisper treat all accents equally?

No. Graham and Roll (2024) found Whisper performs better on American English than on British and Australian accents, and better on native than non-native English speakers.⁶ Nadine's 2026 practitioner benchmark found Whisper Large V3 at 35.05% WER on Nigerian English and Pidgin across 500 samples.⁷ Neither result should be read as a fixed current figure — models retrain — but the direction is consistent across independent measurements.

Why does my dictation change words I actually said?

This is the rewriting error type. The model recognises your phonology but maps it to the standard-spelling variant. Nadine's benchmark found this accounted for a substantial share of non-faithful errors in Whisper on Nigerian speech.⁷ Standard WER counts these substitutions the same as mishearings. Looking at the substitution list — not just the WER score — reveals which words the model reliably changes.

What is word error rate (WER) and how is it measured?

WER is the count of word substitutions, deletions, and insertions between a transcript and a reference, divided by the total words in the reference.¹ A WER of 0.20 means 20 words per 100 were wrong in some way. The jiwer Python library computes WER and prints an alignment showing each substitution, deletion, and insertion — which is more useful than the final number alone.⁹

How do I check the accuracy of dictation for my own voice?

Read a 150–200 word passage you know verbatim into your dictation tool. Normalise both the reference and the transcript identically — lowercase, punctuation stripped — then compute WER with jiwer. The substitution list from the alignment tells you which words the tool reliably changes. Search your past voice notes for those words to find where the model's version replaced yours.⁹

Is the accuracy gap in speech recognition still present in modern models?

The direction is consistent in post-2020 research. Graham and Roll (2024) found Whisper favours American English over British and Australian accents and native over non-native speakers.⁶ Liu et al. (2021) found "significant differences in word error rate across gender and skin tone… at times for all models" in an 846-hour corpus.³ The exact size of the gap for any current commercial system is not established — those require fresh measurement on current systems.

What should I do if my dictation keeps getting specific words wrong?

Run the WER check and read the substitution list. If a small set of words accounts for most errors — a name, a technical term, a word from your dialect — you know the specific failure. For notes that depend on those words, keep the original audio alongside the transcript. A transcript in plain text can be searched and corrected; the audio is the ground truth for anything the transcript rewrites.

The transcript you trust is only as reliable as the voice it was trained on.

A plain-text note is a document you can diff, search, and correct. That does not fix the transcription — but it means the error stays visible rather than getting buried in a format you cannot inspect.

mnmnote.com keeps your notes in open Markdown on your own device — offline, no account required — so a transcript is a file you can search and correct, not a record locked inside an app.

References