Breeze-TTS-2-int4-hqq-g64

Breeze-TTS-2 int4 (HQQ g64, pre-packed)

Weight-only int4 derivative of BreezeBlue/Breeze-TTS-2 (revision c1c8ca18b70b, Apache-2.0). This is not a drop-in transformers checkpoint -- it needs the breeze-tts-runpod loader (int4_tinygemm.py + load_packed.py).

What was quantised

280 bias-free nn.Linears: the 84 in depth_decoder.model.layers[0..11] and the 196 in backbone_model.layers[0..27] (self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj).

Never quantised: the T5Gemma2 text encoder, lm_head, the codebooks head, every embedding, the two projectors, codec_model, and the Qwen3-TTS audio_tokenizer/. Those tensors are here verbatim, in their original dtype and under their original names.

3.25 GiB of bf16 decoder weights become 0.91 GiB of int4 + bf16 scales.

Recipe

Why "pre-packed"

qweight is stored in the portable nibble layout -- exactly the tensor that torch._convert_weight_to_int4pack takes as input -- not in torch's tensor-core-tiled output layout, which is architecture- and version-specific (innerKTiles semantics changed at torch 2.5) and has no CPU equivalent. So a loader does:

packed = torch._convert_weight_to_int4pack(qweight.cuda(), 2)   # milliseconds
mod = Int4Linear(packed, scales_and_zeros.cuda(), K, N, 64)

instead of re-running the quantiser at every boot (0.6 s for RTN g128, 3.4 s for HQQ g64 on an RTX 4090). Shipping HQQ pre-packed is the only reason HQQ is practical here: it has the lower weight error of the two validated variants but was too slow to redo per process.

Quality

The recipe (not this artefact) passed a 320-clip distributional gate on an RTX 4090: 5 texts x {design, clone} x 8 seeds x 4 variants, scored with UTMOS22, DNSMOS, Whisper WER, resemblyzer speaker cosine, pyin F0 and words/s. No int4 variant was distinguishable from bf16 beyond the bf16 seed-to-seed noise floor; HQQ g64 had the lowest clone WER. Speed on a 4090: 47.7 -> 33.5 ms per codec frame (design), 41.3 -> 31.5 (clone); VRAM after warmup 7.53 -> 5.20 GiB.

Files

model.safetensors.index.json is regenerated and maps every tensor to the single model-int4.safetensors. config.json is the upstream one, verbatim -- it still says text_encoder_config.preferred_attn_implementation = "flash_attention_2", which breeze-tts-runpod/model_setup.py:ensure_attn_patch rewrites to sdpa at boot.

Loading

<m>.weight does not exist for the 280 quantised modules, so transformers will report them as missing and report .qweight / .scales_and_zeros as unexpected. See load_packed.py in breeze-tts-runpod/optimization-research/J-packed-checkpoint/ for the two construction routes and the load_packed_int4(model, packed_dir, device) reference implementation. The swap must happen after load_runtime and before Engine.warmup(), which is the CUDA-graph capture.

Licence

Apache-2.0, inherited from BreezeBlue/Breeze-TTS-2. This repository contains only a derived (quantised) form of those weights plus the upstream tokenizer/config files verbatim.