How we optimized omi-medical-edge-1 on CUDA, MLX and CPU

The result

All three rows use unchanged public artifacts, the same scorer and no dictionary, custom vocabulary, contextual bias or transcript correction. The qualified CUDA recipe also powers the hosted omi-medical-edge-1 evaluation route.

WER measures all normalized words. Canonical M-WER aligns the full transcript before counting medical substitutions and deletions. Drug M-WER here is occurrence-safe: it counts missing drug mentions without letting a nearby non-medical alignment shift create a false miss. Throughput is hardware-specific and answers “how fast on this platform,” not which processor is intrinsically more efficient.

On the standing 30-system board, the CUDA and MLX point estimates are the lowest observed WER among the open models we tested. Their WER also came in below the tested configurations of OpenAI GPT-Transcribe at 7.26%, Google Gemini 3.1 Pro Preview at 6.91% and AssemblyAI Universal-3.5 Pro Medical at 6.74%. Those are positions on this benchmark, not universal rankings.

Thirty-nine medical misses were shared by all three runtimes. CPU recovered 11 occurrences missed by CUDA while CUDA recovered five missed by CPU; against MLX the split was 14 versus nine. The paired tests were not significant (p=0.21 and p=0.40). Quantization and decoder rounding flip a few borderline tokens both ways, so the 55/54/49 ordering is not a clinical ranking.

How we tested

We used a 112-row non-board gate to eliminate weak ideas, long recordings to expose memory and boundary failures, and the frozen 1,513-row board for final qualification. That separation matters. The test set did not become a tuning loop.

NeMo could load the checkpoint, but its defaults were not the recipe we had measured. The public adapter now sets precision, attention, decoder behavior, batching, input normalization and whole-record handling before transcription starts.

The selected L4 run scored 6.54% WER and 2.23% M-WER. Short and long slices ran at 216× and 201× realtime respectively.

The most useful Apple correction happened before the encoder. The old MLX frontend placed a 400-sample Hann window at the edge of a 512-point FFT buffer. The training and GPU path center that window. Centering it changed the numerical features sent to the model on every frame.

Cache clearing is a reliability fix, not a transcript cleaner. Without it, two long files in a sequential run produced tens of thousands of unknown tokens. With it, all 1,513 files completed and no transcript contained more than 17 unknown markers.

Bounded attention was the deployability win. Full-attention q8 scored 6.6176% WER. The selected local-attention path scored 6.6473%, a negligible 0.03 percentage-point difference. Local attention ran about 23% faster than the current full-attention arm and cut the reported longest-recording allocation high-water by roughly 71%.

The 89× short and 152× long throughput figures look inverted because the short clips average about nine seconds and pay the per-file setup cost every time. Longer files amortize that fixed cost. We have not separated every component of that setup overhead, so we report the measured result rather than a more specific causal claim.

CPU is the only path here that needs to split long recordings. It decodes recordings up to 240 seconds whole; longer files use overlapping 180-second chunks because full attention in the C++ engine becomes quadratic and impractical. CUDA and MLX use bounded attention and transcribe the recording whole, so they have no equivalent cut to optimize. On CPU, the current runtime searches the 12 seconds before each required boundary and moves the cut to the quietest 300-millisecond window.

The operation is deterministic and uses only the waveform. It does not edit the transcript. On the full board, silence-aware cuts moved WER from 7.17% to 7.10%. The occurrence-safe diagnostic improved from 52 to 49 missing medical mentions and from 21 to 19 missing drug mentions. Canonical M-WER moved the other way, from 2.12% to 2.16%, because its full-transcript alignment marked one additional borderline term as an error. The two measures answer different questions, so we report both.

A separate long-form holdout also improved: 7.32% to 7.25% WER and 0.91% to 0.45% M-WER, with no measurable speed cost. It supports the boundary change; it does not change the platform recommendation.

Run the qualified runtime

For NVIDIA, the CLI is the exact end-to-end path we qualified. It normalizes the input to mono 16 kHz PCM16, pins the checkpoint revision, applies the settings above, sorts files by duration, caps batches at eight files or 900 audio-seconds, and restores caller order.

# NeMo 3.0 is pinned by the package extra
pip install -U "omi-med-stt[nemo]"
omi-med-stt recording.m4a --runtime nemo

Inside Python, the same adapter downloads the Hugging Face checkpoint and applies the qualified GPU settings automatically.

from omi_stt.nemo_runtime import transcribe_nemo

text = transcribe_nemo(
    ["recording.wav"],
    repo_id="omi-health/omi-med-stt-v1",
)[0]

print(text)

On Apple Silicon and CPU, the same package selects the published q8 or q8_0 artifact and the platform recipe described above.

# Apple Silicon
pip install -U "omi-med-stt[mlx]"
omi-med-stt recording.wav --runtime mlx

# Linux / Windows CPU
pip install -U omi-med-stt
omi-med-stt install-cpp --cpp-backend cpu
omi-med-stt recording.wav --runtime cpp

• NVIDIA GPU recipe: exact configuration and release checks.
• NVIDIA adapter source: checkpoint loading, configuration, batching and caller-order restoration.
• Apple MLX q8 recipe: frontend, memory and cache measurements.
• CPU GGUF recipe: silence-aware chunking and statistical caveat.
• Runtime tests: the configuration and control-flow checks that protect the recipes.

Limits of the result

• The quality board is prerecorded English medical audio. It is not a streaming-latency or multilingual evaluation.
• Throughput is hardware-specific and uses explicitly named scopes. Cross-platform speed ratios are descriptive, not an efficiency ranking.
• The board cannot establish a medical ranking among the three runtimes: the paired occurrence comparisons were p=0.21 for CPU versus CUDA and p=0.40 for CPU versus MLX.
• The Apple full-board run used an M4 Max with 64 GB unified memory. It does not validate an 8 GB Mac or a phone.
• Silence-aware boundary selection applies only to CPU recordings that require chunking. CUDA and MLX use whole-record transcription.
• omi-medical-edge-1 is speech-to-text only. It is not clinically validated, and transcripts require review before clinical use.

References and citation

• omi-medical-edge-1 model report
• Medical Speech-to-Text Benchmark
• How to measure medical speech accuracy
• Canonical model weights on Hugging Face

APA: Omi Health. (2026). How we optimized omi-medical-edge-1 on CUDA, MLX and CPU. https://omi.health/research/omi-med-stt-runtime

@misc{omi_med_stt_runtime_2026,
  title  = {How We Optimized omi-medical-edge-1 on CUDA, MLX and CPU},
  author = {{Omi Health}},
  year   = {2026},
  url    = {https://omi.health/research/omi-med-stt-runtime}
}