Medical speech-to-text, tested where it matters.
The same 1,513 clinical clips, scorer and rules across 29 systems. Medical terms and dosage are scored separately.
Three views of the medical record.
Overall wording, medical terminology and dosage events answer different clinical questions. We score all three.
Overall word error
Medical-term error
Dosage F1
29 systems, one board.
Select any accuracy column to reorder the board. Medical-term error is the default because the benchmark ranks medical accuracy first.
| # | System | Type | |||
|---|---|---|---|---|---|
| 1 | omi-medical-1 (live API) | 5.99% | 0.94% | 0.00% | ours · API / self-host |
| 2 | ElevenLabs Scribe v2 | 6.17% | 0.97% | 0.00% | API |
| 3 | Google Chirp 3 | 6.30% | 1.11% | 1.13% | API |
| 4 | VibeVoice-ASR 9B | 7.47% | 1.29% | 1.36% | open |
| 5 | Gemini 3.1 Pro Preview | 6.91% | 1.34% | 0.23% | API |
| 6 | Speechmatics Enhanced Medical | 6.42% | 1.36% | 1.81% | API · medical |
| 7 | Azure MAI-Transcribe-1.5 | 5.97% | 1.43% | 0.90% | API |
| 8 | AssemblyAI Universal-3.5 Pro (Medical) | 6.74% | 1.43% | 1.13% | API · medical |
| 9 | OpenAI gpt-transcribe | 7.26% | 1.43% | 1.36% | API |
| 10 | Soniox STT Async v4 | 6.63% | 1.60% | 3.39% | API |
| 11 | Gemini 3.5 Flash | 7.70% | 1.73% | 0.45% | API |
| 12 | Gladia Solaria-3 | 7.94% | 1.98% | 4.98% | API |
| 13 | OpenAI Whisper-1 | 7.89% | 2.05% | 3.17% | API |
| 14 | omi-medical-edge-1 | 6.64% | 2.16% | 5.43% | ours · open · GPU Parakeet + curated dictionary; on-device unvalidated |
| 15 | Deepgram Nova-3 Medical | 6.98% | 2.19% | 2.26% | API · medical |
| 16 | Reson8 Prerecorded | 6.33% | 2.26% | 6.56% | API |
| 17 | Voxtral Mini Transcribe v2 | 7.82% | 2.47% | 5.66% | open |
| 18 | Whisper Large v3 Turbo | 12.17% | 2.54% | 6.11% | open |
| 19 | Qwen3-ASR 1.7B | 7.01% | 2.79% | 6.11% | open |
| 20 | Qwen3-ASR 0.6B | 7.20% | 3.13% | 7.92% | open |
| 21 | OpenAI GPT-4o Mini Transcribe | 10.02% | 3.24% | 3.39% | API |
| 22 | Parakeet TDT 0.6B v3 | 11.93% | 3.66% | 9.50% | open |
| 23 | Parakeet TDT 0.6B v2 | 12.48% | 3.80% | 8.60% | open |
| 24 | Amazon Transcribe Medical | 6.12% | 4.28% | 1.81% | API · medical |
| 25 | NVIDIA Canary 1B Flash | 12.86% | 4.32% | 13.12% | open |
| 26 | Corti Transcripts | 9.60% | 5.05% | 11.31% | API · medical |
| 27 | Azure gpt-4o-transcribe | 12.52% | 5.57% | 6.33% | API |
| 28 | GCP medical_conversation | 17.70% | 5.57% | 12.44% | API · medical |
| 29 | Google MedASR | 32.21% | 10.83% | 14.48% | open · medical |
Simple enough to audit.
Open any step for the high-level run convention.
1Same clinical audioEvery full-coverage system receives the same 1,513 clips from 57 consultations.
Every system receives the same files, not a provider-specific subset. Full coverage means a transcript for all 1,513 clips. Together they contain 7.2 hours of clinical English and 63.8k reference words.
2Same scorerOne frozen normalization and alignment pipeline scores all transcripts.
Raw transcripts pass through one versioned normalizer and alignment pipeline. Casing, punctuation, medical terms and drug names are handled consistently, and the public scorer records the exact run convention.
3Medical-first rankingMedical terms, drug names and dosage events are measured independently from overall WER.
The board ranks M-WER first, then drug-name error and overall WER. Dosage is a separate event-level board with both precision and recall, so invented doses are penalised. Lower is better for WER, M-WER and drug error; higher is better for dosage F1.
Dataset and scoring definitions
The full-coverage board contains 1,513 clips, 7.2 hours and 63.8k reference words in clinical English. It includes roughly 2,872 medical reference tokens, including 442 drug-name occurrences.
WER measures errors across the whole transcript. M-WER measures errors on the medical lexicon. Drug M-WER isolates drug names. The separate dosage board aligns canonicalised number-and-unit events and reports recall, precision and F1.
The dosage set contains 89 reference events across 63 clips. Omi recovered 86, added no extra doses and missed two after alignment: 96.6% recall, 98.9% precision and 97.7% F1.
Coverage and run convention
“Full coverage” means a system returned a scoreable transcript for every clip in the main board. API systems were called through their available product interfaces; open systems were run under the published harness. The saved model or endpoint name, configuration and scorer version form part of the result.
The primary ranking is M-WER → Drug M-WER → WER. The interactive table can also be ordered by dosage F1, but missing dosage results remain at the bottom rather than being treated as zero.
Interpretation and limitations
- The 0.94% versus 0.97% M-WER difference is about one medical token and is not statistically significant. The two leading systems are a statistical tie on that metric.
- Omi's dosage F1 is significant against every tested competitor under paired cluster bootstrap. Dosage claims always include the underlying 86/89 count and 89-event sample size.
- A result describes the tested model, endpoint and configuration—not every product or deployment a provider offers.
- This board measures prerecorded clinical English. It is not evidence for realtime latency, speaker diarization or multilingual quality.
Reproduce and cite
The public scorer and run instructions are in medical-STT-eval on GitHub. The frozen normalizer, alignment rules and public board generator keep result updates reviewable. Full reruns use the same manifest and row identifiers.
Version history
Changelog
- v6 — August 2026: 29 full-coverage systems, medical-first ranking and the 30-system dosage-event board with precision and F1.
- v5 — July 2026: the 1,513-clip board and frozen v3 scorer became the publication baseline.
- Earlier revisions: retained as historical experiment records; current public claims resolve to v6.
Test Omi on your audio.
25 audio-hours every month. No card required.