Medical Speech-to-Text Benchmark

Our standing benchmark of medical speech-to-text: 28 systems — every major cloud API, the dedicated medical variants, and the leading open models — on identical audio, scored by one open pipeline, ranked by Medical WER: the error rate on the words that matter clinically. Kept current; changelog at the bottom.

From an open test set to a sealed one

Versions 1–4 of this benchmark ran on PriMock57, a public dataset of simulated GP consultations — fully reproducible by anyone, and still archived here. But public benchmark audio has a shelf life: it ends up in training data. We know, because as we developed our own models we trained on part of PriMock57 ourselves — at which point no PriMock-based ranking that includes our models would mean anything, and the same contamination risk applies silently to every vendor on the board.

So v5 moved to a sealed test set, OmiMedSTT-Bench — broader than PriMock, locked and versioned, with zero overlap with our training data. The audio stays private because that is what keeps every score honest, including ours. Everything else is open: the scorer and evaluation code, the convention, the dataset card, and every disclosure. Full-set re-runs available under NDA.

Leaderboard

Ranked medical-first — M-WER, then drug errors, then WER; lower is better. Filter by type, or click a metric to sort.

• Drug M-WER: medical WER on the 442 drug-name occurrences only — the error rate on drug names, same scorer and convention as the overall M-WER column.
• Runaway output — looping or fabricated passages ≥ 3× the reference length — affected only the Gemini models: 33 files for Gemini 3.1 Pro, 88 for Gemini 3.5 Flash. Those files are excluded from their averages; no other system needed exclusions.
• GCP medical_conversation is a verbatim-style model (it transcribes disfluencies our cleaned references drop), so its WER is partly a convention mismatch — disclosed for fairness. On drug names the errors are real.
• VibeVoice-ASR is not deployable at edge latency.
• Run but excluded: Cohere Transcribe and Voxtral Mini v1 were evaluated but returned broken prediction files (empty or garbage output, ~78–157% WER), so they are held out pending a clean re-score rather than ranked on invalid data. Rev AI was disqualified after a partial 50-file probe. None are scored above.

What the board shows

Medical branding is not medical accuracy. The dedicated "medical" products — Amazon Transcribe Medical, GCP medical_conversation, Speechmatics Enhanced Medical — all score worse on medical terms than the top general models. The decisive differences are in drug errors and runaway behaviour, not average WER.

On a separate long-form companion set — 22 complete real consultations (4.1 h), all providers on identical audio, references human-audited — omi-medical-1 leads the next best system by 1.4 percentage points of word accuracy, a bootstrap-certain margin.

Results by scenario

Word and medical-term accuracy by scenario for the top three systems. Best per column shown in green; compare within a row, not across.

Dataset — OmiMedSTT-Bench (sealed)

• 1,513 audio segments · 7.2 hours · 63.8k reference words of English medical speech, from short clinical utterances (median 7 s) to complete 14-minute consultations. The 48 long recordings carry half of all audio hours — the set tests turn-level accuracy and long-form endurance at once.
• Content mix (share of audio hours): doctor–patient consultations 40% · medical vocabulary stress test (scripted utterances with controlled term coverage) 43% · real-world field recordings 7% · everyday-speech control 6% · medical dictation incl. radiology 5%.
• 2,872 medical term occurrences in five categories: drugs (442), symptoms (811), anatomy (806), conditions/diagnoses (332), clinical terms (477).
• Locked and versioned: frozen manifest, hashed artifacts, zero file-level overlap with our training audio. Reference corrections require unanimous agreement of three independent systems plus a human listening to the original audio.

Evaluation framework

One scorer, one convention, identical audio for every system. We publish both strict M-WER (every occurrence must be correct) and information-loss M-WER (did the term reach the transcript at all), for our models and every competitor. Scorer, normalizer and convention are open: github.com/Omi-Health/medical-STT-eval; full-set re-runs available under NDA.

Limitations and what's next

• English only for now; multilingual evaluation is planned.
• Our own set, openly disclosed: we built and hold the test set. The mitigations are published methodology, an open scorer, the audit protocol above, and NDA re-runs. A fully provider-disjoint third-party holdout is planned.
• Speed and cost are tracked separately per-file for all API systems and reported alongside product pages.

Cite this benchmark

APA — Omi Health. (2026). Medical Speech-to-Text Benchmark (v5): 28 Systems Ranked. https://omi.health/benchmark

@misc{omi_stt_benchmark_2026,
  title   = {Medical Speech-to-Text Benchmark (v5): 28 Systems Ranked},
  author  = {{Omi Health}},
  year    = {2026},
  url     = {https://omi.health/benchmark},
  note    = {Medical Word Error Rate (M-WER) leaderboard, sealed test set, open scorer}
}

Changelog

• v5 — July 2026: moved to the sealed OmiMedSTT-Bench set; 28 systems including all hyperscaler medical variants, Speechmatics, Gladia and our own models; medical-first ranking; strict + information-loss metrics published for all systems.
• v1–v4 — July 2025 to April 2026: open PriMock57 benchmark, 42 models. Archived here, rankings frozen, code still open.

Related research

• omi-medical-1 — the flagship model at #1 on this board, available as an API
• omi-medical-edge-1 — the open on-device model at #13, best edge-deployable open medical model measured
• v1–v4 archive — the original open PriMock57 benchmark (42 models, frozen April 2026)
• Clinical SOAP Note Safety Evaluation — 6 models, 300 dialogues, safety-first scoring