Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue

(update — 03 Aug 2025)

What changed in the 03 Aug 2025 refresh

• Extended Whisper-style normalisation: We now strip fillers (“um”), expand contractions (“you’re → you are”), and standardise punctuation/numbers before calculating WER. Most models gained 0.5-1 pp accuracy.

• Extra models & fresh weights: Added Gemini 2.5 Pro / Flash, Voxtral Small, Kyutai

—————————————-

Physicians lose countless hours each week to manual note-taking, so we’re developing an on-device AI-Scribe that can transcribe and summarise entire consultations without sending any data to the cloud. To choose the right speech-to-text engine, we benchmarked 15 open- and closed-source models on PriMock57—a set of 57 simulated GP consultations (5–10 minutes each) recorded by Babylon Health clinicians. We ran the audio files through each model, logged word-error rate (WER), speed, and consistency, and applied chunking only where models crashed on audio longer than 40 seconds. The results surprised us—and they’ll help anyone looking to bring reliable, privacy-first transcription into clinical workflows.

1. Dataset — PriMock57

• 57 simulated GP consultations recorded by seven Babylon Health doctors with role-play patients.

• Reference transcripts were cleaned to plain text for fair Word-Error-Rate (WER) calculation.

• Two recordings (day1_consultation07, day3_consultation03) triggered catastrophic hallucinations on multiple models, so 55 files remain.

• Public repo: https://github.com/babylonhealth/primock57

2. Evaluation framework

• Transcription — a per-model runner saves a .txt and logs processing time.

• Normalisation — extended Whisper-style preprocessing strips fillers ("um"), expands contractions ("you're → you are"), and standardises punctuation/numbers before WER calculation.

• Metrics — scripts compute WER, best/worst file and standard deviation.

• Comparison — results merge into CSV and rankings for plotting.

• Chunking — only applied to Canary-Qwen 2.5 B, Canary-1B-Flash and Phi-4, which break on audio longer than 40 s (30 s chunks with 10 s overlap).

3. Hardware & run types

Local Mac  – Apple M4 Max 64 GB using MLX & WhisperKit

Local GPU – AWS g5.2xl (NVIDIA L4 24 GB) running vLLM and NVIDIA NeMo

Cloud APIs – Groq, ElevenLabs, OpenAI, Mistral, Gemini

Azure     – Foundry Phi-4 multimodal endpoint

4. Results (55 files)

5. Key findings

• Google Gemini 2.5 Pro leads accuracy at 10.8% WER, with Gemini Flash close behind at 12.1%.

• Parakeet-0.6 B on an M4 runs about 5× real-time—great for local use.

• Groq Whisper-v3 (turbo) offers the best price/latency balance in the cloud.

• Chunking rescues Canary, Canary-Qwen & Phi-4 but doubles runtime.

• Apple SpeechAnalyzer is a great hit for Swift apps.

• ElevenLabs Scribe ranks 4th in accuracy but shows high variability (7.0–67.9% range).

• Voxtral-mini beats Canary-Qwen 2.5 B for multilingual long-form.

6. Limitations & next steps

• UK-English only → we’ll use multi-language datasets in the future.

• WER ≠ clinical usefulness → needs more thorough evaluation on medical correctness.

• API cost unmeasured → v2 will include $/hour and CO₂ metrics.

Get in touch

Want to try the on-device AI-Scribe or plug in your own model?

Email [email protected]