Nemotron 3.5 Transcribe VRAM: Which GPU Fits

NVIDIA put Nemotron 3.5 Transcribe on Hugging Face on 28 October 2026. It's an open-weight speech model that turns recordings of up to an hour into text in 40 languages, with speaker labels, word timestamps and translation. Hindi is one of the languages NVIDIA rates "Leading", and it was tested on Hindi-English code-switching too, which is how most of India actually talks.

The name says 30B-A4B. The model card says 34B parameters in total, 4.3B active. NVIDIA's recommended setup is one H100 80 GB. You don't need one. The NVFP4 checkpoint is 20.08 GiB, and the card's own hardware table lists the RTX 5090. Below are the real file sizes of all three checkpoints, the memory the audio adds on top, and the Hindi numbers, so you can pick hardware before you download 67 GB.

Table of contents

• What Nemotron 3.5 Transcribe is
• Checkpoint sizes and the GPUs they fit
• Memory beyond the weights
• Hindi and Hinglish results
• Serving it with vLLM
• Catches before you plan hardware
• FAQ
• Sources

What Nemotron 3.5 Transcribe is

The backbone is a hybrid. In its 52 blocks, 23 are Mamba-2, 23 are mixture-of-experts (128 routed experts, 6 per token, one shared) and only 6 are attention. That split is why the memory maths below comes out so small. The audio goes through an encoder, gets projected to 2,688 dimensions and is fed to the language model with your text instruction. The instruction picks the output style: verbatim with fillers, cleaned-up text, speaker tags, or a translation.

It's an offline model. The card says so in bold: internal 30-second chunking doesn't make it a streaming service. Use it for recorded calls, lectures, podcasts and meetings, not live captions.

Checkpoint sizes and the GPUs they fit

These are the safetensors sizes from the Hugging Face API, not estimates.

Three things follow from the table.

FP8 doesn't fit a 32 GB card. At 32.29 GiB the weights alone are larger than an RTX 5090's memory, before any audio. NVIDIA's smallest FP8 entry is the L40S with 48 GB.

NVFP4 is the only way onto a consumer GPU, and it needs Blackwell. NVFP4 is a Blackwell hardware format. The NVFP4 card says "Blackwell GPUs only", so an RTX 4090 or 3090 with 24 GB can't run it, even though 20 GiB would fit. On those cards you're left with FP8 across two GPUs or a rented H100.

24 GB Blackwell cards are tight. 20.08 GiB of weights leaves under 4 GiB for the CUDA context, the audio encoder's activations and the cache. NVIDIA doesn't list any 24 GB card in its table. I haven't run it on one, so treat it as untested.

On an RTX 5090, weights plus a full-length cache come to about 20.3 GiB, which leaves around 11.7 GiB for activations and the runtime. That's comfortable for one recording at a time.

Memory beyond the weights

On a normal transformer, an hour of audio would eat gigabytes of KV cache. Here it doesn't, because only 6 of the 52 blocks keep one.

From config.json: 6 attention layers, 2 KV heads, head size 128. In BF16 that's 6 x 2 (K and V) x 2 x 128 x 2 bytes = 6 KiB per token. A full 65,536-token context costs 0.375 GiB. The NVFP4 serving command sets --kv-cache-dtype fp8, which halves it to 0.1875 GiB.

The 23 Mamba-2 blocks carry a fixed state instead: 64 heads of 64 with a 128-wide state each. That works out to about 23 MiB per recording in BF16, or 46 MiB if your runtime keeps it in FP32. It doesn't grow with length.

The context budget is where an hour of audio shows up. The card says the encoder runs at roughly 80 ms of audio per token.

• 10 minutes: 7,500 audio tokens
• 30 minutes: 22,500
• 60 minutes: 45,000, which is 68.7% of the context

The remaining 20,536 tokens hold your instruction, any context terms and the transcript itself. At 150 words a minute an hour is 9,000 words, which should fit. Hindi in Devanagari usually takes more tokens per word than English, though, and speaker tags and timestamps add more. If long files come back cut off, split the recording at a pause around the 30-minute mark.

To compare this with other local models, the AI VRAM calculator now has a Nemotron 3.5 Transcribe preset set to the 6 attention layers. Its INT4 setting shows 15.69 GiB of weights, lower than the real NVFP4 file, because NVIDIA keeps parts of the model at higher precision. Use the 20.08 GiB file size when you plan a card. For the general method behind those numbers, see our local LLM VRAM guide.

Hindi and Hinglish results

NVIDIA's language table rates Hindi "Leading", the top of its three levels, with ASR, translation to and from English, multi-speaker ASR and code-switching all supported. Indian English is listed as a supported variant of English. Bengali, Marathi, Tamil and Telugu are at the lowest level, "Supported", which the card says may need fine-tuning.

OIWER is an error rate that doesn't penalise spelling the same Hindi word two valid ways. The mixed-language score is the one to look at if your recordings are Hinglish. About one word in five comes out wrong, roughly double the pure-Hindi error rate. That's fine for searchable notes and first-draft subtitles. It isn't good enough to publish without a read-through.

All of these numbers were measured on BF16. The NVFP4 card says its error rate is on average 3.6% higher in relative terms. Applied to the Hinglish result, 19.28% becomes about 19.97%.

There's also a script catch. NVIDIA's evaluation prompt for Hindi tells the model to "Use Devanagari for Hindi words, never romanized Hindi" and to keep English words in Latin letters. So "kal meeting hai" comes out as "कल meeting है". If you want romanized Hinglish for WhatsApp or YouTube captions, ask for it in the instruction and check the output, because none of the published numbers cover that style.

Serving it with vLLM

NVIDIA's path is vLLM 0.32.0 plus the NeMo Speech main branch, which carries the plugin for this model. The card installs both with uv:

git clone https://github.com/NVIDIA-NeMo/Speech.git nemo-speech
cd nemo-speech
uv venv --python 3.13 .venv
UV_PROJECT_ENVIRONMENT=$PWD/.venv uv sync --extra asr --extra vllm --locked --no-install-project
uv pip install --python .venv/bin/python --no-deps -e .

The NVFP4 serve command from its card, trimmed to the flags that affect memory and audio length:

export VLLM_PLUGINS=nemo_speechlm
export VLLM_MAX_AUDIO_DECODE_DURATION_S=3600
export VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=180
export VLLM_USE_V2_MODEL_RUNNER=0
vllm serve ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 \
  --trust-remote-code --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 --max-model-len 65536 \
  --max-num-seqs 64 --enable-chunked-prefill \
  --use-replayssm --mamba-backend flashinfer --mamba-cache-mode none \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8}' \
  --reasoning-parser nemotron_v3

The card's full command has more flags, including the word-timestamp middleware. Copy it from the card for production. On a single 32 GB card, start with --max-num-seqs well below 64; that value is tuned for a 96 GB RTX PRO 6000.

The server speaks the OpenAI chat format. You send the audio as a base64 audio_url part next to a text instruction, with temperature 0 and thinking off for plain transcription. For readable notes instead of verbatim text, the card's suggested instruction is:

Audio transcription. Language: as spoken. Wording: omit fillers and accidental repetitions; keep the remaining spoken wording unchanged. Formatting: punctuation and capitalization. Output: transcript only.

If your recordings are full of product names or technical terms, add Context terms: and list them. On NVIDIA's earnings-call test, giving local context cut WER from 12.54% to 9.58%.

Catches before you plan hardware

The audio limit defaults to 10 minutes. vLLM decodes 600 seconds of audio unless you set VLLM_MAX_AUDIO_DECODE_DURATION_S=3600. Without it, a one-hour file fails or gets cut, and it looks like a model problem.

The fast numbers aren't in the open release. NVIDIA's headline speed, 2,238 seconds of audio per second on one RTX PRO 6000, uses NVFP4, DFlash2 and the optimised vLLM inside the Nemotron Transcribe NIM. The card says the open-source vLLM path is expected to be roughly two times slower. That's still fast for one person's recordings, but don't size a business on the headline figure.

FP8 and NVFP4 aren't interchangeable across GPU generations. NVIDIA lists FP8 only on Ada (L40S) and newer, and NVFP4 only on Blackwell. Its A100 entry is BF16 only, which means 80 GB.

No speaker identity. Speaker labels tell speakers apart within one file. The card says they don't identify real people, so "Speaker 1" in one call isn't the same person as "Speaker 1" in the next.

FAQ

Can I run it with Ollama or LM Studio? Not from NVIDIA's checkpoints. They're NeMo-format safetensors that need the NeMo Speech plugin. NVIDIA's table lists GGML on the RTX 5090 and RTX PRO 6000, but it publishes no GGUF. Community GGUF and MLX conversions appeared on Hugging Face within days. None of the error rates above were measured on them, and I haven't tested whether Ollama or LM Studio loads the audio side.

Is it better than Whisper for Hindi? NVIDIA's headline tables compare it with Qwen3-ASR-1.7B and Parakeet, not Whisper. On the 7-language multilingual set it scores 4.62% mean WER against Qwen3-ASR's 6.49%. There's no Whisper Hindi number on the card, so I can't give you a fair head-to-head.

Does it need internet once downloaded? No. It runs fully offline on your own GPU, which helps when the recordings are client calls you'd rather not upload.

Sources

• NVIDIA, NVIDIA-Nemotron-3.5-Transcribe-30B-A4B model card (huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B, read 4 November 2026): release dates, parameters, architecture, language matrix, Hindi and code-switching results, hardware table, vLLM setup, the 80 ms token rate, the NIM speed figure.
• NVIDIA, NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 model card (read 4 November 2026): Blackwell-only note, the 3.6% relative error increase, the NVFP4 serve command.
• NVIDIA, config.json in the BF16 repo: block types, KV heads, head size, Mamba-2 state dimensions, expert counts.
• Hugging Face model API for the BF16, FP8, NVFP4 and DFlash2 repos (read 4 November 2026): safetensors file sizes.
• KV cache, Mamba state, GiB conversions and context budget are calculated by ToolMintX from the values above.