parakeet-tdt-0.6b-v3

parakeet-tdt-0.6b-v3 — half-precision and 4-bit checkpoints

Reduced-precision builds of NVIDIA's nvidia/parakeet-tdt-0.6b-v3, packaged for direct GPU loading. Three artifacts, same model:

The two .nemo files are pure casts of the fp32 weights — identical to upstream apart from the conversion. Restoring from either loads straight onto a CUDA GPU without ever materialising an fp32 copy, which is faster to start (~13 s vs ~22 s) and roughly halves peak load-time VRAM.

The NVFP4 directory is not a cast: 4-bit scales are chosen by calibrating on real audio, so it ships pre-quantized. It halves VRAM again at no accuracy cost this evaluation can detect, and costs 2.4× the decode latency.

Which one?

• bf16 — the default. Fastest, and what the accuracy of the others is measured against.
• fp16 — if your GPU predates Ampere, where bf16 is not merely slower but unsupported. On a card that does both, the two are statistically indistinguishable and fp16 is marginally closer to fp32 numerically.
• nvfp4 — only when you are short of VRAM. It is the smallest by a wide margin and the slowest of the three; it buys memory, not throughput.

This repository redistributes the model under its original CC BY 4.0 license. See the upstream model card for architecture, training data, supported languages, evaluation, and intended use.

Usage

bf16 / fp16 (NeMo)

import nemo.collections.asr as nemo_asr
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("lightware-dev/parakeet-tdt-0.6b-v3",
                       "parakeet-tdt-0.6b-v3-bf16.nemo")   # or -fp16.nemo
model = nemo_asr.models.ASRModel.restore_from(ckpt, map_location="cuda")
model.eval()
print(model.transcribe(["audio.wav"]))

NVFP4

Needs nvidia-modelopt alongside NeMo. The loader lives in lightware-dev/blurt as server/nvfp4.py:

import os
from huggingface_hub import snapshot_download
from server import nvfp4          # from github.com/lightware-dev/blurt

root = snapshot_download("lightware-dev/parakeet-tdt-0.6b-v3",
                         allow_patterns="parakeet-tdt-0.6b-v3-nvfp4/*")
model = nvfp4.load_snapshot(os.path.join(root, "parakeet-tdt-0.6b-v3-nvfp4"),
                            device="cuda")
print(model.transcribe(["audio.wav"]))

In blurtd itself this is just PARAKEET_DTYPE=nvfp4.

The load path builds the module graph on the CPU, replays the quantization recipe onto it, then reads the packed tensors straight onto the GPU — so the device never holds a bf16 copy. That is the entire point: quantizing at startup instead would require the GPU to hold the full bf16 model first (peaking at 2.53 GB), which a card small enough to need 4-bit cannot do.

What NVFP4 is here

NVFP4 is E2M1 — four bits per weight (1 sign, 2 exponent, 1 mantissa, so eight representable magnitudes) — grouped in blocks of 16, each block carrying its own E4M3 fp8 scale. The block scale is what makes four bits survivable: precision is allocated locally, so a block of small weights is not crushed by a large outlier elsewhere in the row. It is also why the packed model is 0.49 GB rather than the 0.27 GB a naive 4-bit count would suggest.

Only the 217 encoder Linear layers are quantized — 533M of 627M parameters (86%). The convolutions, LSTM decoder and joint network stay bf16, and so do all activations (W4A16). The joint network's vocabulary projection is deliberately excluded: it runs inside the TDT decode loop, where an extra unpack per step would cost latency and feed the argmax directly.

Contents of the directory:

No pickles. torch.load reconstructs arbitrary Python objects, so loading one is equivalent to running its author's code — a poor property for a file fetched at service startup. Weights use safetensors (a length-prefixed JSON header over raw tensor bytes, structurally unable to carry code), the recipe is JSON, the config is YAML. The single class the recipe names is resolved by attribute lookup against a modelopt.* allowlist, never called.

Evaluation

All figures below come from one corpus of 208 clips / 25.8 minutes: 100 LibriSpeech test-clean utterances, the same utterances degraded with white noise at 10 dB and 5 dB SNR, babble at 5 dB and a near-clipping loud variant, plus 8 synthetic clips. Greedy TDT decoding, WER after text normalisation, on an RTX 5090. Differences against bf16 are paired bootstrap estimates — paired because both models decode identical audio, so per-utterance noise cancels instead of being counted twice.

NVFP4 was calibrated on 62 clips drawn from different LibriSpeech rows than the 208 scored ones, asserted disjoint at build time. Calibrating on the evaluation set would tune the quantizer to its own test.

NVFP4 vs bf16

NVFP4 is not better than bf16. Its point estimate is lower and its babble-noise slice looks notably better, but the confidence interval spans −0.69 to +0.30 pp — that is noise landing favourably. At 208 clips and a ~3 % base rate the corpus cannot resolve a difference below roughly 0.3 pp, so the honest claim is "no cost we can measure", not "identical".

What is visible: four bits changes the exact text of 14 % of clips while leaving the aggregate untouched, and pushes the encoder output about 5× further from fp32 than bf16 does. The greedy decoder absorbs most of that drift.

Per slice:

The babble and synthetic slices move in NVFP4's favour and the white-noise slices against it, on 25 and 8 clips respectively. Do not read a denoising story into that — these are small slices and the aggregate CI already says the corpus cannot separate the two models.

Memory

Measured per precision in a fresh process, loading the published artifact the way a service would:

The load-time peak is the figure that decides whether a card can run the model at all, and it is where shipping pre-quantized weights pays: 0.78 GB against 2.53 GB. Quantizing a bf16 checkpoint at startup would produce the same 0.51 GB of weights but still need 2.53 GB to get there.

Load time is a wash. Of the ~21 s, only 0.3 s is device work — 0.2 s to read 489 MB of packed tensors onto CUDA and 0.1 s to bind them. The rest is CPU overhead that should not need to exist: NeMo cannot construct this model on the meta device (ConformerEncoder.__init__ calls .item()), so a full module graph is built and zero-filled, and ModelOpt's restore replays the compression on those throwaway weights rather than installing the packed layout directly.

Speed

NVFP4 decodes 2.4× slower than bf16 (RTF 0.0316 vs 0.0134). Encoder forward time is 95.6 ms against 27.0 ms.

The reason is that Parakeet's encoder is bound by kernel launches, not arithmetic. Encoder forward time barely responds to how much audio it is given:

Sixteen times the audio for 6 % more time means the GPU spends its life launching kernels across 24 conformer layers and 220 Linears. Quantization makes the multiplies cheaper — the part that was never the bottleneck — while adding an unpack per layer. It lands net negative. At RTF 0.032 the model is still ~30× faster than real time, which is imperceptible for dictation, but NVFP4 should not be chosen expecting throughput.

fp16 vs bf16 vs fp32

fp16 and bf16 produced identical transcripts on 207 of 208 clips. The WER difference between them is +0.026 pp (95 % bootstrap CI [0.000, +0.077], p = 0.74) — statistically indistinguishable, and exactly zero on real speech.

fp16 tracks fp32 roughly 8× more closely than bf16 does at the activation level, which is what you would expect from its 10 mantissa bits against bf16's 7. No overflow was observed: zero non-finite activations anywhere, including on the deliberately near-clipping loud slice. The narrower fp16 exponent range is not a practical problem for this model at inference.

Other quantizations evaluated, and why they are not here

The same corpus and method were used to test five other post-training schemes. None is published — this table is here so the choice of NVFP4 is checkable rather than asserted.

Two results shaped the choice:

• Weights tolerate four bits; activations do not. NVFP4 W4A4 is the only variant with a statistically significant regression (CI excludes zero, p = 0.006), and it is worst on babble noise (10.43 % against bf16's 8.29 %). Quantizing weights only is what keeps 4-bit free.
• Four bits is a better deal than eight here. NVFP4 saves 38 % more memory than INT8 weight-only for the same undetectable accuracy cost, which inverts the usual expectation that fewer bits means more risk.

Nothing was faster than bf16 at any width, for the launch-bound reason above.

Caveats

• English-only corpus. These figures say nothing directly about the other 24 languages the base model supports. For the pure casts (bf16/fp16) there is a strong prior that a weight cast behaves the same everywhere; for NVFP4 that prior is weaker, because the scales were calibrated on English speech. If you use it on another language, measure.
• 208 clips bounds the resolution. Differences below roughly 0.3 pp are beyond what this corpus can see.
• NVFP4 is tied to its tooling. Built with ModelOpt 0.45.0 / PyTorch 2.12.1 / NeMo 2.7.3; recipe.json records the ModelOpt version. Cross-version restore is untested — treat the snapshot as rebuildable from source rather than archival.
• Speed figures are RTX 5090 (sm_120). The quality figures carry across hardware; the timings do not.
• Both .nemo files were cast from the upstream fp32 weights — the fp16 one is not a bf16 → fp16 round trip, which would inherit bf16's coarser mantissa for no benefit. NVFP4 was quantized from the bf16 checkpoint.

Provenance

• Base model: nvidia/parakeet-tdt-0.6b-v3
• bf16 / fp16: fp32 weights cast, re-saved as .nemo (701 float tensors, 627,090,582 parameters, unchanged key set)
• NVFP4: 217 encoder Linear layers quantized to NVFP4 W4A16 with NVIDIA TensorRT Model Optimizer 0.45.0, calibrated on 62 LibriSpeech clips held out from the evaluation set, compressed to packed 4-bit and serialized as safetensors + JSON
• License: CC BY 4.0 (inherited from the base model)
• Built and evaluated with: lightware-dev/blurt — scripts/build_bf16_ckpt.py, scripts/build_nvfp4_snapshot.py, scripts/make_eval_corpus.py and scripts/compare_precision.py

Each snapshot build records the transcripts it produced in memory; the loader is verified by reproducing them exactly from disk (20 / 20), since a mis-bound scale buffer still yields fluent English.

Previously published as lightware-dev/parakeet-tdt-0.6b-v3-bf16 and lightware-dev/parakeet-tdt-0.6b-v3-fp16; both are now consolidated here.