NVIDIA's Nemotron 3.5 Transcribe Turns Speech-to-Text Into a Prompting Problem

Voice AI has spent two years insisting that the next interface is speech. Most of that promise is still delivered by a chain of separate models: one system hears, another works out who is talking, a third translates, and a language model turns the result into something a person or an application can use. NVIDIA's release on October 28, 2026 bets on a flatter design. Nemotron 3.5 Transcribe is an open-weight speech model that accepts a recording plus a natural-language instruction and returns whichever output the task needs -- a plain transcript, a speaker-attributed transcript, a translation, answers about the audio, or text with personal data stripped out.

The checkpoint is a 30-billion-parameter mixture of experts that activates roughly 4 billion parameters per token, published on Hugging Face under an open license that permits commercial use. It shipped alongside Granary v2, an expanded open speech dataset and curation pipeline, and fine-tuning recipes built on NeMo Speech. Together they form a path from data preparation through adaptation to a served endpoint, which is more than most model drops of this size bother to include.

What NVIDIA actually shipped

The capability list is the part that matters for anyone designing a speech product, because it describes several jobs that usually require several models.

The name and the parameter count disagree slightly, and that is worth understanding rather than glossing over. The checkpoint is called 30B-A4B, but the model card lists 34B total parameters once the audio encoder is counted, with about 4.3B active. The active figure describes per-token compute during generation, not the memory needed to hold the weights. Anyone planning a deployment should size hardware against the total checkpoint and its precision, not against the active count. The release offers BF16, FP8, and NVFP4 variants precisely because that memory difference is significant at the edge.

A prompt instead of a pipeline

The conventional production stack for speech analytics is a sequence of specialists. An ASR model produces a transcript. A diarization model assigns speakers. A translation model handles a second language. An LLM summarizes or extracts answers. A separate pass redacts personal information. Each handoff adds an interface, another deployment dependency, and another chance to lose timing or speaker context between stages.

Nemotron collapses much of that into one request. The same recording can yield different products depending on the instruction:

• Transcribe this recording and label each speaker.
• Translate this audio into English.
• What decisions were made in this meeting?
• Return a transcript with PII redacted.

That flexibility is not just convenience. Because the decoder can attach speaker labels while generating the transcript, the acoustic and speaker information stays inside a single generation process instead of being reconciled afterward by a separate alignment step. The same holds for formatting: asking for verbatim output with disfluencies, or for clean written form with numbers and punctuation normalized, is a prompt-level choice rather than a different model.

Contextual biasing is the other practical lever. General-purpose recognizers tend to mangle drug names, product SKUs, and customer surnames. The model accepts a short list of candidate terms, with an important instruction to use them only when the audio supports it. NVIDIA's own example for a clinical setting supplies terms such as HbA1c and metformin, requests standard medical spelling, and asks for conventional written number formats. That framing matters: a term list that overrides the acoustics will manufacture errors just as confidently as a model without context.

The trade-off is real. Collapsing a pipeline into one model means one error profile instead of five, and one place to look when output is wrong. It also means less ability to swap a component because a specific language, accent, or domain performs poorly. Evaluation shifts from per-stage metrics to end-to-end quality, which is closer to what users experience but harder to attribute.

Inside the checkpoint

Nemotron 3.5 Transcribe is a speech-augmented language model. Two encoders sit in front of an autoregressive backbone: a transformer ASR encoder for acoustics and a speaker encoder whose cues reach the decoder alongside the audio features. The backbone is a hybrid of Mamba state-space layers, Transformer attention, and sparsely routed expert modules, so only a fraction of the weights run for each generated token.

That design explains the efficiency claims. In NVIDIA's reported serving numbers, one RTX PRO 6000 Blackwell Server Edition GPU processes up to 2,238 seconds of audio per second using NVFP4 with speculative decoding, and returns the transcript of a 30-second clip in 0.13 seconds. Those optimizations are not all in the open-source release yet; the vLLM integration shipped in NeMo Speech on release day is described as roughly twice as slow and intended for development rather than peak throughput.

Two architectural details matter for application design. First, native multi-token prediction supports speculative decoding, which is how the throughput figures above are reached. Second, reasoning mode is configurable through the chat template. Transcription and translation want temperature zero and reasoning off; audio question answering is recommended at a higher temperature with reasoning on. Treating those two modes as the same workload is a reliable way to get slower, worse results.

What the benchmarks say, and what they do not

NVIDIA reports strong numbers across several independent boards, and the pattern is consistent: the model is competitive on accuracy while using far fewer active parameters than the open-weight models it ties.

The two English figures are a useful reminder about leaderboard reading. The 5.33% average comes from the leaderboard's standard English track; the 4.71 mean comes from a nine-dataset grouping in the model card. Different dataset mixes, different normalization, different numbers. Neither is wrong, and neither predicts performance on a specific call center recording in a specific accent.

The parameters-versus-accuracy story is the more durable claim. Artificial Analysis placed the model on its open-weight frontier of word error rate against active parameters, matching a 24B-active competitor with roughly one-sixth the per-token compute. For teams paying for inference by the minute, active parameters are the cost driver, so that comparison is the one to watch.

Speaker attribution deserves separate attention because it is usually a bolted-on component. NVIDIA reports the lowest diarization error rate among 14 systems on David AI's 21-language benchmark, slightly ahead of its own dedicated diarization model, and first place on VoiceArena's 14-system comparison for both in-person and online sessions. On VoiceArena's concatenated minimum-permutation metric, deletions accounted for 8.32% of reference words against a benchmark median near 31%, which suggests the model is dropping less speech than typical systems. Independent private-set results point the same way: on Cekura's Ocular WER set the model scored 2.48%, ahead of the proprietary APIs on that board.

None of this removes the need for local evaluation. Public benchmarks cannot tell you how a model handles your accents, your overlapping speech, your brand names, or your telephony codecs. They tell you whether a model is worth the cost of that evaluation.

The offline constraint is the real boundary

One limitation shapes every architectural decision around this model: it is offline. It processes recorded audio, not live streams. For meeting transcription, contact-center QA, media captioning, medical dictation, and searchable audio archives, that is exactly the right trade. Accuracy benefits from seeing the whole recording, and a one-hour context window plus word-level timestamps covers most batch workflows without client-side chunking.

For voice agents, it is not enough on its own. A conversational agent needs partial transcripts within a few hundred milliseconds, and rival releases in the same month targeted precisely that gap: ElevenLabs shipped Realtime speech-to-speech models with sub-second time to first audio, Mistral shipped Voxtral 3, and Salesforce released Launa-Omni. NVIDIA's model competes on a different axis, and the practical pattern is a two-pass design: a streaming recognizer handles the live turn, and a batch model like this one reprocesses the completed recording for the accurate, speaker-attributed, well-formatted version that gets stored, searched, and summarized.

Latency in the batch path is good enough to make that pattern comfortable. Independent benchmarking on a hosted endpoint measured a speed factor of about 75x, transcribing one minute of audio in under a second at roughly $2.00 per 1,000 minutes. DeepInfra lists $0.002 per minute, or $0.12 per hour of audio, served through an OpenAI-compatible speech-to-text endpoint with zero data retention.

Granary v2 and the adaptation path

The dataset is the quieter half of the release and possibly the more useful one for teams with domain problems. Granary v2 covers 463,062 hours across 48 languages from 16 source corpora, up from 25 languages in the first version. Its pipeline does the unglamorous work that determines whether a speech model is trainable at all: segmentation, two-pass inference, hallucination filtering, language identification checks, punctuation and casing restoration, translation pair generation, and automatic quality scoring.

The most interesting design choice is that each clip gets three aligned transcript forms: a text-normalized verbatim version that preserves fillers, an inverse-normalized version that keeps disfluencies but writes numbers conventionally, and a cleaned inverse-normalized version with fillers removed. That makes punctuation, number formatting, and disfluency handling explicit targets rather than emergent behavior. If your application needs verbatim transcripts for compliance and clean transcripts for search, this is the data structure that teaches a model to produce both on request.

Fine-tuning results show what the adaptation path is worth. On a benchmark of recordings from Indian doctors and medical students, NVIDIA reports the following progression on the full set:

On held-out speakers, fine-tuning cut WER from 22.44% with the strongest prompt to 12.17%, a 45.8% relative reduction, and beat both an open medical ASR specialist and a widely used general-purpose model on the same speakers and scorer. The recipe trains the speech encoder and projection layer while applying low-rank adapters to the language model, and the whole workflow took about 14 GPU-hours on a single GB300. That is a realistic budget for a mid-sized team with a domain problem.

The trade-offs are equally instructive. After medical fine-tuning, LibriSpeech test-clean WER rose from 1.54% to 1.69%, and the specialist medical model kept its edge on brand-name drug recognition. Domain adaptation narrows a model and costs you something elsewhere. Anyone who ships a fine-tuned checkpoint without re-running general evaluation will eventually find that cost in production.

Running it without surprises

Serving options are broad for a model of this size: vLLM through NeMo Speech for GPU servers, NeMo-Speech.cpp for local execution, and hosted endpoints for teams that would rather not operate anything. A minimal API call against a hosted endpoint looks like this:

python import os from openai import OpenAI

client = OpenAI( api_key=os.environ['DEEPINFRA_API_KEY'], base_url='https://api.deepinfra.com/v1/openai', )

with open('consultation.mp3', 'rb') as audio: result = client.audio.transcriptions.create( model='nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B', file=audio, prompt='Transcribe this consultation in the original language and label each speaker. ' 'Possible terms: HbA1c, metformin. Use these terms only when supported by the audio.', response_format='verbose_json', timestamp_granularities=['word', 'segment'], )

for segment in result.segments: print(f'[{segment.start:.1f}s - {segment.end:.1f}s] {segment.text}')

A few practices prevent most of the frustration people hit with speech models. Keep the system prompt empty for transcription and translation, and put the instruction in the user message; the model card is explicit about this. Ask for the output shape you need in the prompt rather than post-processing it, since verbatim, cleaned, timestamped, and speaker-labeled outputs are all supported requests. Spot-check names, speaker changes, and any answers produced by audio question answering against the recording, because audio QA is trained in English even when transcription is not. And check the language support matrix before promising coverage, since support differs by task: 40 languages for transcription does not mean 40 for diarization or translation.

What to take away

The interesting thing about Nemotron 3.5 Transcribe is not that it tops a leaderboard. It is that it treats a transcript as one of several possible outputs from a single recording rather than the fixed product of a fixed pipeline. For meeting notes, call review, media archives, and clinical documentation, that reframing removes real integration work and keeps speaker and timing context intact.

The gaps are equally clear. It is a recorded-audio model in a year when voice agents are the fastest-moving part of the category, so it belongs beside a streaming recognizer rather than instead of one. Its best results come with adaptation, and adaptation carries measurable regressions. And benchmark numbers gathered from different dataset mixes are a starting point for evaluation, never a substitute for it. The teams that get the most from this release will be the ones that treat the prompt as a first-class interface, test on their own audio early, and fine-tune only after the prompts stop improving.

References

• TechCrunch: Voice AI in vogue: Nvidia releases new open transcription model
• Hugging Face Blog: NVIDIA Nemotron 3.5 Transcribe and Granary v2
• Model Card: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B