parakeet-tdt-0.6b-v2-MLX-Mixed-3bit4bitBF16

Parakeet TDT 0.6B v2 — MLX Mixed 3-bit/4-bit/BF16

A sensitivity-guided mixed-precision MLX conversion of the English-only nvidia/parakeet-tdt-0.6b-v2, built from the unquantized FP32 mlx-community/parakeet-tdt-0.6b-v2 checkpoint.

The 0.349 GB safetensor uses group-64 affine quantization for 265 eligible matrices: 100 attention matrices at 3-bit and 165 encoder, feed-forward, and pointwise-convolution matrices at 4-bit. The eight predictor and joint matrices remain BF16, as do normalization, depthwise-convolution, bias, and other ineligible weights.

It is 25.8% smaller than the 0.470 GB external encoder-INT4 model while matching its macro WER on a secondary 387-row domain panel. On the full 6,206-utterance English gate, it is the throughput-focused option: 51.3% faster and 1.04% smaller than the separately published Unified mixed-precision model, with 0.06 point higher macro WER.

Full three-collection evaluation

Measured locally on Apple Silicon with MLX and public parakeet-mlx inference. The gate uses the complete LibriSpeech test-clean (2,620), complete LibriSpeech test-other (2,939), and complete FLEURS en_us test (647) splits. Every artifact scored all 6,206 utterances, totaling 12.51 audio hours, with zero skipped references. Macro WER gives each collection equal weight.

Against FP32, this model trades 0.07 macro-WER point for an 85.90% reduction in weight storage. There is no universal winner among the compressed models: Unified has the cautious quality edge, while this TDT release is the throughput choice.

Secondary meeting and financial-speech panel

An earlier seed-2026 comparison sampled up to 100 examples from LibriSpeech clean, LibriSpeech other, AMI, and Earnings-22. AMI contributed 87 valid rows, for 387 matched utterances. It remains useful domain evidence, but is not the headline quality gate.

The TDT candidate and external INT4 model are a practical quality tie on this panel. Their 0.01-point difference is not an accuracy claim; the useful result is retaining that quality class with 0.121 GB less weight storage.

Allocation and rejected candidates

The calibration set contains 12 FLEURS en_us validation clips selected with seed 31415; it is disjoint from the final test rows. Four-layer attention, feed-forward, and pointwise-convolution regions plus the predictor and joint were independently fake-quantized against a BF16 teacher.

The accepted allocation is:

• encoder attention in layers 0–15 and 20–23: affine Q3;
• encoder attention in layers 16–19: affine Q4;
• all encoder feed-forward matrices: affine Q4;
• encoder pre-projection and all pointwise convolutions: affine Q4;
• predictor embedding, four LSTM matrices, and three joint matrices: BF16.

The broader screen rejected several attractive-looking size points. Uniform Q3 reached 0.272 GB but 11.88% macro WER on the seed-42 screen. Uniform Q4 reached 0.349 GB but 8.17%. A 0.323 GB Q3-attention/Q3-FFN composition reached 8.35%, showing that low isolated calibration drift did not always compose safely.

Provenance and reproducibility

• NVIDIA source revision: ae9ad07059c7c739ffaf932226a8fe64ae2620b0
• MLX Community FP32 revision: 8ae155301e23d820d82aa60d24817c900e69e487
• External INT4 revision: 35bbe15bda247cada6f3e54b9156cfddf0ad7b51

The final checkpoint was rebuilt from the pinned FP32 source in a clean output directory and reproduced the evaluated weights byte-for-byte:

model.safetensors SHA-256
278c556f6be491dcc1c2972de1ab2f20ae22f6ee47caec474154a9d607c3500c

The exact allocation is included as quantization_recipe.json. The compatibility loader is included as parakeet_unified_mlx.py; conversion code, sensitivity results, rejected profiles, and matched benchmark evidence are maintained in yuting1214/model-lab.

Usage

Install the public runtime and loader dependencies:

pip install parakeet-mlx huggingface_hub pyyaml librosa soundfile

from huggingface_hub import snapshot_download
from parakeet_unified_mlx import load_parakeet_mlx

model_dir = snapshot_download(
    "MarkChen1214/parakeet-tdt-0.6b-v2-MLX-Mixed-3bit4bitBF16"
)
model = load_parakeet_mlx(model_dir)
result = model.generate("speech.wav")
print(result.text)

Run the script from the downloaded repository directory, or place parakeet_unified_mlx.py on PYTHONPATH before importing it.

Scope and limitations

• This is an English-only model. It is not a replacement for multilingual ASR.
• Measurements use offline full-context TDT inference and greedy decoding.
• The included loader is required because current public parakeet-mlx does not instantiate per-module mixed bit widths from this saved recipe by itself.
• RTFx is machine-, runtime-, and thermal-state dependent. Treat the matched local comparison as directional and remeasure on your deployment hardware.
• Accuracy varies with accents, domains, noise, and recording conditions. Evaluate on your own target distribution before deployment.

License

The upstream model is released under CC-BY-4.0. This quantized derivative keeps that license and attribution. See the upstream model card for details.