Nemotron 3.5 Transcribe vs Qwen3-ASR: what leaderboards miss (2026 test)
Nemotron 3.5 Transcribe vs Qwen3-ASR, tested 2026-10-29 to 2026-10-30 on public audio: on read speech the two are close, but Nemotron by default wrote numbers as words on all three runtimes we tried, adding 6.9 to 7.0 points of raw WER on earnings calls (13.1% raw against 6.2% with numbers spelled out, hosted). In our 4-bit Mac build, an instruction change came with skipped passages in 3 of 5 excerpts from one meeting. Disclosure: we sell a transcription service.
We compared two open speech recognition models, NVIDIA Nemotron 3.5 Transcribe 30B-A4B and Qwen3-ASR-1.7B, on public test sets only, through a hosted inference API (both models) and with NVIDIA's official weights run locally on an Apple M4 Pro laptop. The point is not a new leaderboard row. It is what changes once you run these models yourself: number formatting, instructions, runtimes, and languages the model was not built for.
| What we measured (as tested 2026-10-29 to 2026-10-30) | Qwen3-ASR-1.7B | Nemotron 3.5 Transcribe |
|---|---|---|
| Earnings calls, raw WER (24 windows, hosted API) | 7.1% | 13.1% |
| Earnings calls, WER with numbers spelled out on both sides | 6.6% | 6.2% |
| Numbers returned as digits (digit-group ratio, hosted API) | 0.98 | 0.00 |
| FLEURS, nine word-scored languages pooled (990 clips) | 7.3% hosted | 9.0% hosted, 8.1% local 4-bit |
| Mandarin Chinese, character error rate (130 clips) | 4.9% hosted | 9.1% hosted, 7.3% local 4-bit |
| LibriSpeech test-clean (80 clips) | 1.71% hosted | 1.45% hosted |
| Meeting excerpts with a skipped passage of 15+ words (5 excerpts, one AMI meeting) | 0 of 5 hosted | 0 of 5 hosted; 3 of 5 in our local 4-bit build with the written-numbers instruction |
| Peak memory on the laptop, batch 1 | 4.4 GiB (MLX 8-bit build) | 19.7 GiB (MLX 4-bit), 24.4 GiB (NVIDIA NVFP4) |
All error rates are lower-is-better. Full tables with 95% confidence intervals, the per-item scores and the method are in the paper (PDF).
How accurate is Nemotron 3.5 Transcribe compared with Qwen3-ASR?
On read speech, close, with large differences in a few languages. On the FLEURS test set, scored with numbers spelled out on both sides so that number formatting does not count, Qwen3-ASR-1.7B through the hosted API had a pooled word error rate of 7.3% (95% CI 6.7 to 7.9) across nine word-scored languages, against 9.0% (8.2 to 9.8) for Nemotron on the same API and 8.1% (7.4 to 8.9) for NVIDIA's official Nemotron weights run locally at 4-bit, on the same 990 clips tested 2026-10-29 to 2026-10-30.
| Language | Metric | Clips | Qwen3-ASR-1.7B, hosted | Nemotron, hosted | Nemotron, local 4-bit |
|---|---|---|---|---|---|
| Arabic | WER | 120 | 14.2% (11.8 to 16.7) | 14.0% (11.7 to 16.7) | 11.5% (9.4 to 14.1) |
| German | WER | 120 | 5.3% (4.0 to 6.9) | 7.0% (5.1 to 9.1) | 5.6% (3.9 to 7.4) |
| English (indicative) | WER | 10 | 8.5% (1.6 to 21.6) | 8.5% (0.9 to 24.1) | 10.0% (2.6 to 24.1) |
| Spanish | WER | 130 | 3.5% (2.6 to 4.5) | 3.3% (2.3 to 4.6) | 2.9% (2.0 to 3.9) |
| French | WER | 120 | 5.3% (4.1 to 6.7) | 5.5% (4.3 to 6.8) | 5.4% (4.2 to 6.6) |
| Hindi | WER | 130 | 11.7% (9.8 to 13.6) | 13.4% (10.0 to 17.3) | 11.0% (8.3 to 14.2) |
| Indonesian | WER | 120 | 4.9% (3.4 to 6.8) | 16.3% (13.9 to 18.7) | 17.1% (14.5 to 19.9) |
| Japanese (indicative) | CER | 10 | 5.0% (2.7 to 7.9) | 0.9% (0.0 to 2.0) | 2.0% (0.8 to 3.4) |
| Korean | CER | 130 | 4.1% (2.4 to 6.5) | 3.9% (2.6 to 5.5) | 3.7% (2.4 to 5.4) |
| Portuguese | WER | 120 | 6.4% (4.4 to 8.8) | 5.9% (4.1 to 8.2) | 5.5% (3.6 to 7.9) |
| Russian | WER | 120 | 8.2% (6.3 to 10.4) | 9.4% (7.2 to 12.1) | 8.6% (6.5 to 11.1) |
| Mandarin Chinese | CER | 130 | 4.9% (3.4 to 6.7) | 9.1% (7.1 to 11.3) | 7.3% (5.4 to 9.4) |
| Pooled, nine word-scored languages | WER | 990 | 7.3% (6.7 to 7.9) | 9.0% (8.2 to 9.8) | 8.1% (7.4 to 8.9) |
Numbers in brackets are 95% confidence intervals from a paired bootstrap over clips. English and Japanese had only ten clips each in this three-way subset, so treat those rows as indicative.
- Where the local Nemotron build led: Arabic (2.7 points lower than Qwen3-ASR, interval 1.2 to 4.1), Portuguese and Spanish (smaller margins, intervals touching zero for Spanish), and Japanese on ten clips.
- Where Qwen3-ASR led: Indonesian by 11 to 12 points on both Nemotron runtimes, and Mandarin Chinese.
- English read speech: on LibriSpeech test-clean (80 clips) the rates were 1.71% for Qwen3-ASR and 1.45% for Nemotron on the hosted API, and 1.39% for the NVFP4 build; the intervals overlap, so this sample does not separate them.
Why does Nemotron write numbers as words, and what does it cost on earnings calls?
Without an instruction asking for written numbers, Nemotron 3.5 Transcribe returns "twenty twenty one" where a reference transcript has "2021". On 24 Earnings-22 earnings-call windows of about 160 seconds each, tested 2026-10-29 to 2026-10-30, it did this on all three runtimes we tried: the hosted API, our local 4-bit build with the mlx-audio default instruction, and NVIDIA's NVFP4 checkpoint on mlx-audio. That alone added 6.9 to 7.0 points of raw word error rate. With numbers spelled out on both sides, the gap to Qwen3-ASR disappears.
| Model and runtime | Raw WER (95% CI) | WER, numbers spelled on both sides (95% CI) | Digit-group ratio |
|---|---|---|---|
| Qwen3-ASR-1.7B, hosted API | 7.1% (6.0 to 8.3) | 6.6% (5.6 to 7.8) | 0.98 |
| Nemotron, hosted API | 13.1% (11.1 to 15.1) | 6.2% (5.3 to 7.1) | 0.00 |
| Nemotron, local 4-bit, default instruction | 13.5% (11.4 to 15.5) | 6.5% (5.7 to 7.4) | 0.00 |
| Nemotron, NVIDIA NVFP4, default instruction | 13.8% (11.7 to 16.0) | 6.9% (6.0 to 7.9) | 0.00 |
| Nemotron, local 4-bit, written-numbers instruction | 11.5% (9.3 to 13.8) | 6.6% (5.6 to 7.6) | 0.36 |
| Nemotron, NVIDIA NVFP4, speaker-attributed instruction | 12.4% (10.2 to 14.7) | 7.1% (6.2 to 8.0) | 0.25 |
The digit-group ratio is the number of digit groups in the output divided by the number in the reference: 1 means numbers come back as digits as often as the reference writes them. Nemotron's model card documents an instruction that asks for numbers in their standard written form. On the hosted API we could not change the instruction the service applies, so its output never contained digits. Locally, that instruction moved the ratio to 0.36, not to 1, and it came with the problem in the next section.
Can an instruction change make a speech model skip words?
In our build, it coincided with exactly that. With the written-numbers instruction, our local 4-bit Nemotron build lost runs of 15 or more consecutive words in 3 of 5 excerpts from one AMI meeting (EN2002a), tested 2026-10-29 to 2026-10-30: the longest run was 155 words, and the other two excerpts lost 32 and 17 words. Nothing in the output flagged the gap. With the mlx-audio default instruction, the same build on the same five excerpts had no such runs and the lowest error rate in the comparison after NVFP4.
| Meeting excerpt (about 150 s each) | Qwen3-ASR-1.7B, hosted | Nemotron, hosted | Nemotron 4-bit, written numbers | Nemotron 4-bit, default | Nemotron NVFP4, default |
|---|---|---|---|---|---|
| 1 | 14.7% | 11.1% | 10.2% | 12.0% | 9.6% |
| 2 | 14.0% | 17.3% | 55.2% (runs of 155 and 30 words) | 14.2% | 12.4% |
| 3 | 13.8% | 12.6% | 19.0% (a run of 32 words) | 12.4% | 12.4% |
| 4 | 9.7% | 17.3% | 17.8% (a run of 17 words) | 10.9% | 9.3% |
| 5 | 14.5% | 10.3% | 7.1% | 11.1% | 11.7% |
| All five | 13.4% | 13.7% | 22.1% | 12.1% | 11.2% |
Five excerpts from a single meeting are far too few for a confidence interval, so read this as an association we observed, not a measured property of the model or of NVIDIA's instruction. We did not reproduce it on a reference runtime or at higher precision. The same 4-bit build with the same instruction had 0 of 24 Earnings-22 windows with such a run, and NVIDIA's NVFP4 build with the default instruction dropped a run of 22 words once, in 1 of its 29 long windows. On the three meeting excerpts where every NVFP4 run completed, the speaker-attributed task had a WER of 8.3% against 14.3% for Qwen3-ASR on the hosted API (n too small for a CI). If you run Nemotron locally on long audio, check for silent gaps: compare the output length with the audio length, or align the transcript.
How much memory does Nemotron need on a Mac?
About 18 to 24 GiB per stream in our tests, against about 4 GiB for Qwen3-ASR-1.7B. These are peak memory figures on a 48 GB Apple M4 Pro at batch 1, measured 2026-10-29 to 2026-10-30.
| Build | Peak memory | Notes |
|---|---|---|
| Qwen3-ASR-1.7B, MLX 8-bit | 3.2 to 4.4 GiB | 70 clips |
| Nemotron, MLX 4-bit (official weights, converted by us) | 18.2 to 19.7 GiB | all clips completed |
| Nemotron, NVIDIA NVFP4 on mlx-audio main | 23.0 to 24.4 GiB | word timestamps on; with the speaker-attributed task, 2 of 5 meeting excerpts failed (one alignment error, one GPU timeout) |
| Nemotron, MLX 8-bit (converted by us) | 33.3 to 34.4 GiB | 3 of 4 meeting excerpts completed, then a GPU timeout |
Two deployment notes from the same days. Converting the official weights to MLX on the GPU hit a Metal timeout, so we converted on the CPU. Word timestamps for the official checkpoints landed in mlx-audio's main branch on 2026-10-29 (pull request 1092) and were not yet in a release when we tested. We do not report speed: after the 8-bit run's GPU timeout the laptop's throughput was not stable, and one laptop is not a serving benchmark.
What happens with languages the model does not support?
Persian and Malay are not among the languages listed on the Nemotron 3.5 Transcribe model card. Through the hosted API, Nemotron still returned a transcript for every clip, with no error and no warning, and its error rate was 34.3% on Persian and 36.7% on Malay, against 25.4% and 10.3% for Qwen3-ASR-1.7B on the same clips (tested 2026-10-29).
| Language | Clips | Qwen3-ASR-1.7B, hosted | Nemotron, hosted |
|---|---|---|---|
| Persian (indicative) | 27 | 25.4% (20.9 to 30.5) | 34.3% (29.7 to 39.0) |
| Malay (indicative) | 29 | 10.3% (6.5 to 14.3) | 36.7% (30.9 to 42.4) |
If your audio can be in any language, check the language against the model card before sending it; the response will not tell you.
How did we measure?
- Audio. Public test sets only: FLEURS test (CC BY 4.0), LibriSpeech test-clean with forced-aligned word times (CC BY 4.0), five excerpts of about 150 seconds from AMI meeting EN2002a, headset mix (CC BY 4.0), and 24 windows of about 160 seconds from three Earnings-22 calls (CC BY-SA 4.0). No customer or private audio was used.
- Systems. Qwen3-ASR-1.7B and Nemotron 3.5 Transcribe through the same hosted inference API, which received each clip's language as a hint; NVIDIA's official Nemotron weights (Hugging Face revision 9d8cd6b7) converted by us to MLX 4-bit and run with mlx-audio 0.6.3, once with a written-numbers instruction (paraphrased from the model card; the exact string passed in the run was not logged) and once with mlx-audio's default instruction; NVIDIA's NVFP4 checkpoint on mlx-audio main (commit 5f42a925) with the default and speaker-attributed tasks. The local runs received no language hint. We do not name the hosted service.
- Scoring. Word error rate (character error rate for Chinese, Japanese and Korean) after lowercasing and removing punctuation and fillers. "Numbers spelled on both sides" converts digits to words in the reference and the output before scoring, so it measures recognition rather than number formatting.
- Skipped passages. A run of 15 or more consecutive reference words deleted in the alignment of one long window. We report it for the meeting and earnings windows only.
- Statistics. 95% intervals from a paired percentile bootstrap over clips, 2,000 resamples, fixed seed; pooled rows resample within each language. Where n is 5 or fewer we print "n too small for a CI".
What are the limitations?
- The skipped passages were not reproduced on a reference runtime or at higher precision. They come from our own 4-bit MLX conversion, five excerpts of one meeting, and one written-numbers instruction (paraphrased from the model card; the exact string passed in the run was not logged). They may come from the quantization, the port, the instruction, or their combination; we cannot tell which from this data. The NVFP4 build, with the default instruction, also dropped a run of 22 words in 1 of its 29 long windows.
- Hosted versus local is confounded. The hosted runs use the precision the service reports (bf16) and an instruction we cannot see or change; our local build is 4-bit with a different instruction and no language hint. The Mandarin gap between the two runtimes (9.1% against 7.3%) is a difference we measured, not one we can attribute to a cause.
- Small samples. The meeting results rest on one AMI meeting, and English and Japanese have ten clips each in the three-way FLEURS subset. A larger two-way sample of the hosted runs, including Czech, Dutch, Italian and Vietnamese at about 30 clips each, is in the paper's appendix.
- One point in time. Hosted services and open-source runtimes change quickly. These results describe what we measured on 2026-10-29 and 2026-10-30.
Where can I get the data?
- Per-table results with intervals (CSV): open-asr-models-2026-10.csv
- Full results, model revisions and per-item scores (JSON): open-asr-models-2026-10-results.json
- The paper (PDF): Open speech recognition models beyond the leaderboard
The files contain scores and item identifiers only, not transcripts. For our earlier public-audio studies, see the YouTube auto-caption accuracy study and the companion on speakers and timing.
Conflict of interest: Transcribe.so sells a transcription service and ran this evaluation itself. Nobody at NVIDIA or the Qwen team funded or reviewed it. We tested both models on equal terms and report the results that favour each.
Frequently asked questions
Is Nemotron 3.5 Transcribe more accurate than Qwen3-ASR?
It depends on the language and on how you run it. On FLEURS with numbers normalized, Qwen3-ASR-1.7B was lower on pooled word error rate (7.3% against 8.1% for Nemotron's official weights at 4-bit), Nemotron's local build was lower on Arabic, and Qwen3-ASR was much lower on Indonesian and Mandarin Chinese, tested 2026-10-29 to 2026-10-30.
Why does Nemotron 3.5 Transcribe write numbers as words?
Its default output spells numbers out. The model card documents an instruction that asks for numbers in standard written form. Through the hosted API we used, that instruction could not be set, and on 24 earnings-call windows the spelled-out numbers added 6.9 to 7.0 points of raw word error rate.
Does Nemotron 3.5 Transcribe skip words?
In our local 4-bit build with the written-numbers instruction, 3 of 5 excerpts from one meeting lost runs of 15 or more words, the longest 155 words. We saw none with the default instruction on the same excerpts. This is an association from five excerpts, not reproduced at higher precision.
How much memory does Nemotron 3.5 Transcribe need on a Mac?
In our tests on a 48 GB M4 Pro at batch 1, peak memory was 18.2 to 19.7 GiB for a 4-bit MLX build and 23.0 to 24.4 GiB for NVIDIA's NVFP4 checkpoint, against 3.2 to 4.4 GiB for an 8-bit Qwen3-ASR-1.7B build.
Which languages does Nemotron 3.5 Transcribe not support?
The model card lists the supported languages; Persian and Malay are not among them. Sent anyway through a hosted API, they came back with error rates of 34.3% and 36.7% and no warning.
Which hosted API did you use?
We do not name vendors. The finding that matters is general: a hosted wrapper can fix the instruction and change the output format, so measure the runtime you will actually use.