Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue

Physicians lose countless hours each week to manual note-taking. At Omi, we're building an on-device AI-Scribe that transcribes and summarises entire consultations without sending patient data to the cloud. Choosing the right speech-to-text engine is the foundation of that product — so we built the most comprehensive medical STT benchmark we could find, and we keep it current.

What's new in v4

We now rank models by Medical WER (M-WER) — a metric that only counts errors on clinically relevant words (drugs, conditions, symptoms, anatomy, clinical procedures). Standard WER treats "yeah" and "amoxicillin" as equally important. M-WER doesn't — and the ranking looks very different as a result. We also report Drug M-WER separately, because misspelling a drug name is a patient safety issue, and drug names turn out to be 2–5x harder than other medical terms for almost every model.

• 42 models benchmarked across cloud APIs (Google, OpenAI, ElevenLabs, Soniox, AssemblyAI, Deepgram, Mistral, Microsoft, Cohere, Groq) and local/open-source (VibeVoice, Qwen3-ASR, Parakeet, Whisper variants, NVIDIA Canary, Gemma 4, and more).
• Google Gemini 3 Pro leads the leaderboard at 2.65% M-WER.
• Microsoft VibeVoice-ASR 9B is the first open-source model to genuinely compete with Gemini-tier cloud APIs (#3, 3.16% M-WER).
• Qwen3-ASR 1.7B is the best small open-source model — 4.40% M-WER at ~7s per file on an A10 GPU.
• Custom text normalizer replacing Whisper's default, after we discovered two bugs that were quietly inflating WER by 2–3% across every model in the industry. Open-source and drop-in.

Leaderboard

Ranked by Medical WER (M-WER). Lower is better. Dataset: PriMock57 — 55 simulated GP consultations, ~80,500 words of British English medical dialogue.

Dataset — PriMock57

• 57 simulated GP consultations (5–10 minutes each), recorded by seven Babylon Health doctors with role-play patients.
• 55 files used after removing two that triggered catastrophic hallucinations on multiple models.
• ~80,500 words of British English medical dialogue.
• Reference transcripts cleaned to plain text for fair WER calculation.
• Public dataset: babylonhealth/primock57

Evaluation framework

Every model is run through the same pipeline: transcribe the audio, log processing time, normalise the output with our custom text normalizer, and compute WER, M-WER, Drug M-WER, and per-file statistics. Models that crash on long audio get chunking with overlap (typically 30s chunks with 10s overlap and LCS merging). We run on a mix of Apple Silicon (M4 Max), NVIDIA GPUs (T4, L4, A10, H100), and cloud APIs directly. No quantisation on local models unless explicitly noted — we benchmark full precision for fairness.

Full evaluation code: github.com/Omi-Health/medical-STT-eval

Choosing a medical STT engine

Best accuracy, cloud API

Google Gemini 3 Pro or 2.5 Pro. Both sit below 3% M-WER and handle long-form conversation natively.

Strong accuracy without Google

Soniox, AssemblyAI Universal-3 Pro (with medical-v1 domain), or Deepgram Nova-3 Medical. All three are genuinely competitive with Gemini-tier on M-WER, and Deepgram is the fastest cloud API on the board at 13s/file.

Best open-source, accuracy-first

Microsoft VibeVoice-ASR 9B. First open-source model to compete with Gemini on medical audio. Requires ~18GB VRAM (L4/A10 is enough; won't fit on T4) and is slow — 97s/file even on H100 — because it's a 9B LLM processing audio as tokens rather than a purpose-built speech model.

Best open-source, cost/speed-balanced

Qwen3-ASR 1.7B. ~14x faster than VibeVoice for only ~1.3 points more M-WER. Requires compute capability ≥ 8.0 (A10 or better; not T4).

On-device, Apple Silicon

Parakeet TDT models are the fastest option, but be aware their Drug M-WER is weak — 22% for the 0.6B v3 — which may not be acceptable in a clinical context.

What to avoid for medical transcription

Google MedASR (despite being medical-specific, it's built for single-speaker dictation, not conversations — 49.66% M-WER), Facebook MMS-1B-all (multilingual phonetic vocab, 54% M-WER), and current quantised versions of VibeVoice (aggressive quantisation kills drug-name accuracy).

Limitations and what's next

• UK English only — the PriMock57 dataset is British English. We'll extend to multi-language datasets in future updates.
• WER ≠ clinical usefulness — M-WER is a meaningful step toward clinical relevance, but human-graded clinical correctness is the next frontier.
• API cost unmeasured — a future update will include $/hour and latency SLO metrics for cloud APIs.
• Medical vocabulary is extensible — our 179-term list is a starting point; contributions welcome via the repo.

Cite this benchmark

APA — Omi Health. (2025). Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue. https://omi.health/research/stt-benchmark-primock

@misc{omi_stt_benchmark_2025,
  title   = {Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue},
  author  = {{Omi Health}},
  year    = {2025},
  url     = {https://omi.health/research/stt-benchmark-primock},
  note    = {Medical Word Error Rate (M-WER) leaderboard, 42 models}
}

Related research

• Medical STT Benchmark v5 (current) — the sealed successor to this benchmark: 28 systems on 7.2 h of multi-scenario medical audio
• omi-medical-edge-1 — our on-device 0.6B medical speech-to-text model, benchmarked against 28 open and cloud systems
• Clinical SOAP Note Safety Evaluation — 6 models, 300 dialogues, safety-first scoring
• Omi-Sum 3B — open-source clinical model for SOAP note summarization