parakeet-tdt-0.6b-v3-gguf

parakeet-tdt-0.6b-v3 — calibrated GGUF quants (Starling engine)

Calibrated block-quantized GGUFs of nvidia/parakeet-tdt-0.6b-v3 (FastConformer

• TDT transducer, 25 European languages, CC-BY-4.0), built with Starling's in-tree quantization pipeline: llama.cpp-style quant levels (Q8_0 → IQ2) whose block scales are weighted by an importance matrix collected on real speech across all 25 supported languages (FLEURS train, 48 clips/language), verified by WER with bootstrap confidence intervals.

Runtime note: these files follow the Starling GGUF tensor contract and run on the native starling-serve binary / libstarling_ggml engine from the starling repository. They are not llama.cpp/whisper.cpp/parakeet.cpp GGUFs. (The converse works, though: the starling engine also loads community parakeet GGUFs — the cstr/parakeet.cpp and transcribe.cpp dialects — directly, via its loader compat layer.)

Files

All levels ≥ Q4 were verified indistinguishable from f32 on clean and 5 dB-noised English audio and on 300-clip German/English test sets; the sub-2-bit IQ file trades tail-language accuracy for size (details below). The 640-row joint/LSTM linears automatically stay at Q8_0 (block-size rule) in every level.

Phone file: parakeet-tdt-0.6b-v3-q4pw-fullimx.gguf

Every linear Q4_0, including the conformer pointwise convolutions (which all other files keep at F16); the 640-wide joint/LSTM and the prediction embedding Q8_0. Q4_0 repacks into arm64's i8mm GEMM and maps losslessly onto the Starling Vulkan fast engine. Built from the F32 conversion with parakeet-q4pw-fullimx.recipe + parakeet-full.imx (both in this repo) and --shrink-f16.

Pixel 10 Pro, Vulkan fast engine, vs q4_k_m-shrink16: load 1.88 vs 2.30 s, 22 s clip 2.06 vs 2.26 s, 74 s clip 6.91 vs 7.69 s, peak RSS 778 vs 1180 MB, about −15 % energy per transcription; identical fixture transcripts.

SHA-256: 6975b243883552a41abac56fd4021aef8ec57489de8fb8854c464fb41a551e26

Measured quality (FLEURS test, mean WER [95% bootstrap CI])

25-language mean: f32 14.0 → q2_k 16.9 → iq2_xxs 26.4 (48 clips/language; the iq2_xxs gap is concentrated in lt/lv/sl/mt/hu/sk). Choose by deployment: q4_k_m if quality matters, q2_k for even multilingual coverage at 574 MB, iq2_xxs only for English-first deployments at 325 MB.

Calibration-data notes (measured, not assumed): importance-matrix statistics saturate at roughly a dozen clips per language (5× more calibration audio changed the matrix by <1% and WER not at all); the calibration language mix is not a measurable lever for English/German once a matrix exists (EN-only vs 25-language matrices are statistically indistinguishable on EN/DE at 300 clips); what matters is calibrated-vs-uniform (uniform Q2_K degrades to 30%/17% WER where the calibrated build matches f32) and the bit width itself.

Usage

hf download scholzmx/parakeet-tdt-0.6b-v3-gguf \
  parakeet-tdt-0.6b-v3-q4_k_m.gguf --local-dir ./models

starling-serve --model parakeet --gguf ./models/parakeet-tdt-0.6b-v3-q4_k_m.gguf --port 8181

Provenance

• Base weights: nvidia/parakeet-tdt-0.6b-v3 (CC-BY-4.0)
• Conversion: starling convert_parakeet_gguf.py (F32 base)
• Calibration audio: FLEURS train split, 24 of the 25 languages × 48 clips (Slovenian's shard was unavailable during collection; the measured language-mix insensitivity makes this immaterial)
• Quantizer: starling starling-quantize (ggml block quantization + imatrix)
• Evaluation: starling wer_quant.py, FLEURS test (300 clips EN/DE, 48/language elsewhere)

Quantization pipeline and full result tables: docs/quantization.md in the starling repository.