Qwen3.8-27B-PARO-int5

Qwen3.8-27B-PARO-int5

Qwen3.8-27B with ParoQuant rotations on uniform asymmetric int5 weights, built for AMD RDNA4 (Radeon AI PRO R9700, gfx1201) and served through a W5A8 fp8-WMMA path in a patched vLLM. It is the fidelity-first sibling of Qwen3.8-27B-PARO-MXFP4: 4.2x lower KL divergence for +2.6 ms/step and 3 GB more on disk.

• Weights: uniform asymmetric int5, group size 128 along K, fp16 group scales and zero points (5.25 bits/weight effective). Stored as "int5-bitplane": qweight/qzeros hold the low four bits in the int4 AWQ packing, and qweight_hi/qzeros_hi are [K, N/32] int32 fifth-bit planes.
• Rotations: the learned pairwise Givens rotations (pairs, theta, 8 layers of 64 disjoint pairs per 128-channel group) and pre-inverted channel_scales from z-lab/Qwen3.8-27B-PARO (ParoQuant, ICLR'26), reused unchanged — rotations trained from scratch on a consumer-RAM calibration budget come out as identity past layer ~10.
• How it was made: bf16 base weights x z-lab's channel scales, rotated with z-lab's rotations, quantized one-shot to int5 g128 RTN, then a stage-2 fine-tune of the weights and group scales under the int5 grid (rotations frozen). Inference identity: y = ((x * channel_scales) R^T) dequant(Q)^T.
• Config: quantization_config.quant_method = "paroquant", bits = 5, format = "int5-bitplane", group_size = 128, krot = 8. lm_head and embeddings are bf16, as in the source checkpoints. Tokenizer, chat template and generation config are Qwen's.

Why five bits

The int4 kernel feeds the fp8 WMMA the signed code c - 8, which is exact in e4m3. With 5-bit codes, c - 16 spans -16..15 and every integer in that range is also exact in e4m3 — so the GEMM algebra, the zero-point fold, the per-token activation quant, the rotation-stream producers and the split-K / A-tiled bands all carry over unchanged. Only weight staging differs: low nibbles keep the existing word layout and the fifth bit rides in a byte-per-(slot, lane) plane in the same fragment order. int6 does not have this property (codes to ±32 are not exact in e4m3) and would need an int8-WMMA rewrite.

At decode the 5-bit kernel costs 1.20-1.26x the 4-bit kernel at M<=8 — exactly the 1.235x byte ratio, so it is purely bandwidth-bound with no unpack penalty.

Loading

This is not loadable by stock transformers or stock vLLM: paroquant is a vLLM quantization plugin that fuses the rotation with the activation quant and runs the int5 x fp8 WMMA GEMM. It ships in radiance-vllm-mxfp4 (paroquant/, PAROQUANT.md), which builds the kernels in-container on ROCm for gfx1201:

./setup-paroquant.sh
MODEL_DIR=Qwen3.8-27B-PARO-int5 MODE=prod SPEC=7 \
  RADIANCE_PQ_I8=1 RADIANCE_PQ_PG=1 RADIANCE_PQ_ZPE=1 \
  ./paroquant/run_paroquant.sh

Those three flags select the configuration all the numbers below were measured with: int8 per-group activations (I8 + PG) and the zero-point epilogue (ZPE). Other hardware would need the rotation prologue and GEMM ported; the format itself is plain int5 plus the rotation tensors, so a dequantize-and-rotate reference is a few lines.

Results (2 x R9700, TP=2, fp8 KV, DFlash2-FP8 drafter, SPEC=7)

KL is KL(bf16 || candidate) over wikitext, 96 x 500-char chunks, top-256, with the bf16 reference served on the same stack — a reference collected on a different image charges quantization ~0.018 nats that belong to the serving stack's own numerics.

Where the fidelity comes from, cumulatively: fp16 group scales instead of e8m0 shared exponents (-32% KL), the fifth bit (0.0285 -> 0.0155), the stage-2 fine-tune (-> 0.0126), and int8 per-group activations instead of e4m3 per-token (-> 0.0097). The last one matters more than it looks: per-token int8 was the wrong granularity, not the wrong format, and per-group int8 lands below the weights-only RTN number.

Choosing between this and PARO-MXFP4

Task accuracy does not separate them — GSM8K is 97.4-97.8% for every variant, inside noise at 500 questions. The separation is distributional fidelity: at 0.0100 vs 0.0419 nats this checkpoint stays markedly closer to the bf16 base's full output distribution, which is what matters for logprob-sensitive work, draft-model acceptance, and long agentic chains where small per-token divergences compound. If you want maximum prefill throughput and KV headroom instead, take PARO-MXFP4.