NVIDIA’s Nemotron Speech staff has launched Nemotron 3.5 ASR. It’s a 600M-parameter streaming Computerized Speech Recognition (ASR) mannequin. A single checkpoint transcribes 40 language-locales in actual time. Punctuation and capitalization are in-built natively. The mannequin ships as open weights on Hugging Face. The license is OpenMDW-1.1. The structure is a Cache-Conscious FastConformer-RNNT.
What’s Nemotron 3.5 ASR
Nemotron 3.5 ASR extends nvidia/nemotron-speech-streaming-en-0.6b to many languages. It provides prompt-based language-ID conditioning to the bottom mannequin. That lets one 600M-parameter checkpoint cowl 40 language-locales. No per-language mannequin or model-swapping is required.
The mannequin targets two workloads. The primary is low-latency streaming for reside audio. The second is high-throughput batch transcription. Output is production-ready textual content with correct casing and punctuation. No separate punctuation-restoration step is required.
How Cache-Conscious FastConformer-RNNT Works
The mannequin has two foremost items. The primary is a Cache-Conscious FastConformer encoder with 24 layers. FastConformer is an environment friendly evolution of the Conformer structure. It makes use of linearly scalable consideration. The second piece is an RNNT (Recurrent Neural Community Transducer) decoder. RNNT emits textual content body by body as audio streams in.
The “cache-aware” design is the effectivity lever. Buffered streaming re-processes overlapping audio home windows at each step. That repeats the identical work and provides delay. This mannequin caches encoder self-attention and convolution activations as a substitute. It reuses these cached states as new audio arrives. So every audio body is processed precisely as soon as, with no overlap. Compute and end-to-end latency each drop, with out an accuracy penalty.
The Latency Knob: att_context_size
One inference setting controls the latency-accuracy tradeoff. It’s the consideration context dimension, att_context_size. Smaller context emits textual content sooner however sees much less future audio. Bigger context raises accuracy at increased latency.
The identical checkpoint covers the complete vary. Settings map to chunk sizes of 80ms, 160ms, 320ms, 560ms, and 1.12s. For instance, [56,0] provides an 80ms ultra-low-latency mode. The [56,13] setting provides 1.12s for highest accuracy. Groups decide the working level at inference time, with no retraining.
Language Detection and Protection
The 40 language-locales embody English, Spanish, German, and French variants. In addition they cowl Arabic, Japanese, Korean, Mandarin, Hindi, and Thai. A number of different European and Nordic languages are included too.
Language conditioning works two methods. Setting target_lang to a recognized locale often provides the most effective accuracy. Setting target_lang=auto lets the mannequin detect the language itself. In auto mode, it emits a language tag after terminal punctuation. One deployment can then transcribe mixed-language site visitors. No separate language-ID part is required.
Comparability
| Product | Firm | Entry | Native streaming | Language protection | Reported latency | Pricing mannequin |
|---|---|---|---|---|---|---|
| Nemotron 3.5 ASR | NVIDIA | Open weights (OpenMDW-1.1), self-host; hosted on DeepInfra | Sure — cache-aware FastConformer-RNNT | 40 language-locales | 80ms–1.12s, configurable at inference | Free to self-host; usage-based by way of host |
| Whisper large-v3 | OpenAI | Open weights (MIT), self-host; API | No — offline/batch | ~99 languages | Not streaming-native | Self-host free; API ~$0.006/min (batch) |
| Nova-3 | Deepgram | Closed API; on-premise/self-host (enterprise) | Sure — streaming + batch | Multilingual; +10 monolingual languages added Jan 2026 | Low-latency streaming (reported sub-300ms) | ~$0.0077/min (Nova-3 Monolingual, PAYG) |
| Common-3 Professional Streaming | AssemblyAI | Closed API (EU endpoint accessible) | Sure | 6 languages: English, Spanish, French, German, Italian, Portuguese | Sub-300ms (official); first partial ~750ms | Utilization-based (PAYG) |
| Scribe v2 Realtime | ElevenLabs | Closed API | Sure | 90+ languages (99 per ElevenLabs) | ~150ms (p50) | ~$0.28/hour |
| Ursa / streaming | Speechmatics | API + on-premise + edge | Sure — streaming + batch | 50+ languages with automated identification | Extremely-low latency (positioned) | Enterprise/utilization |
Advantageous-Tuning Outcomes
As a result of the weights are open, groups can fine-tune for a language, area, or accent. NVIDIA revealed a labored instance on Greek and Bulgarian. It fine-tuned the bottom checkpoint with the identical Cache-Conscious FastConformer-RNNT recipe. Every clip carried a target_lang tag for language conditioning. Coaching information got here from public corpora, together with Granary, Widespread Voice, and FLEURS.
Outcomes had been measured as WER on held-out FLEURS, on the 80ms setting. Greek WER fell from 35 to 24, a 32% relative enchancment. Bulgarian fell from 22 to fifteen, a 31% relative enchancment. These are uncooked WER percentages on the lowest-latency streaming mode. NVIDIA notes that evaluating at deployment latency, on held-out information, provides trustworthy numbers.
Strengths and Issues
Strengths:
- One 600M-parameter checkpoint covers 40 language-locales, chopping deployment sprawl.
- Cache-aware streaming processes every body as soon as, reported at 17x buffered concurrency on an H100.
att_context_sizetunes latency from 80ms to 1.12s at inference, with no retraining.- Punctuation, capitalization, and
autolanguage tagging are in-built. - Open weights enabled a 31–32% relative WER drop on Greek and Bulgarian after fine-tuning.
Issues:
- The mannequin handles English, however NVIDIA recommends its devoted English mannequin for English-only use.
- The 80ms mode trades some accuracy for the bottom latency.
- Japanese and Korean use CER, so cross-language error comparisons want care.
- Throughput figures are measured on H100, so outcomes on different GPUs will differ.
- The manufacturing NIM with gRPC streaming is introduced, however not but launched.
Key Takeaways
- NVIDIA’s Nemotron 3.5 ASR is an open-weights (OpenMDW-1.1), 600M-parameter streaming mannequin transcribing 40 language-locales from one checkpoint.
- Its Cache-Conscious FastConformer-RNNT design processes every audio body as soon as, reported at 17x the concurrent streams of buffered approaches on an H100.
- Latency is configurable from 80ms to 1.12s at inference by way of
att_context_size, with no retraining. - A brief fine-tune minimize FLEURS WER 32% on Greek (35→24) and 31% on Bulgarian (22→15), on the 80ms setting.
- It’s self-hostable and streaming-native, not like closed APIs (Deepgram, AssemblyAI, ElevenLabs) or offline Whisper.
Marktechpost’s Visible Explainer
NEMOTRON 3.5 ASR
1 / 10
Curated for AI engineers by Marktechpost — practitioner-first protection of AI & ML.
Take a look at the Model weights. Additionally, be at liberty to observe us on Twitter and don’t neglect to hitch our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Have to associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so forth.? Connect with us


