mnma_qwen3.8_27b_nvfp4

mnma Qwen3.8-27B NVFP4 (Minima)

A fully W4A4 build of Qwen3.8-27B by post-training quantization alone: every one of the 496 backbone linear layers — all 48 Gated DeltaNet (GDN) layers and their gate projections included — quantized to NVFP4, plus calibrated FP8 KV-cache scales. No quantization-aware training, no distillation — calibration only.

Early public 4-bit builds of this model kept the GDN block at 8/16-bit on the assumption that recurrent-state layers are fragile under quantization. Our accompanying paper shows the opposite — the architecture's own gating and delta-rule correction make the recurrent half the easy half to quantize — and this checkpoint is the proof: it matches BF16 within seed noise on every task we measured while being the smallest and fastest-prefill recipe in the cohort the paper compares.

What exactly is quantized

Built with llm-compressor (NVFP4 scheme + kv_cache_scheme fp8/tensor/static), calibrated on a frozen 128-sample × 32K-token set. Text-only Qwen3_5ForCausalLM extraction (no vision tower).

Global scales are harmonized. vLLM fuses in_proj_qkv+z and in_proj_b+a into single NVFP4 GEMMs with one global scale each, while llm-compressor calibrates them per module (up to 2.8× apart). Serving an unharmonized checkpoint silently mis-scales the GDN decay/write gates. This checkpoint has the fused groups rewritten to shared global scales with the ratio folded into the per-block scales, so it serves correctly out of the box.

Serving

Requires vLLM ≥ 0.27 on hardware with native NVFP4 (SM120 / Blackwell):

vllm serve minima-ai/mnma_qwen3.8_27b_nvfp4 --kv-cache-dtype fp8

--kv-cache-dtype fp8 picks up the calibrated k/v scales shipped in the checkpoint.

Evaluation

Measured under one serving regime (vLLM 0.27.1, TP=1, one 96 GB RTX PRO 6000, FP8 KV cache), against the BF16 model served identically:

No task score is CI-separated from BF16. Task rows were measured on the scale-free twin of this checkpoint (identical weights; a KV-cache ablation showed KV precision moves no task score for either model); PPL, RULER, and throughput were measured on this checkpoint directly. The perplexity gap (+0.73 @4K, +0.15 @32K) shrinks with context — quantization error does not accumulate in the recurrent state; see the paper for the mechanism study.

Notes

• Long-context perplexity of the base model rises inside long requests (BF16 shows the same 6.95 → 10.35 pattern); compare models only under matched serving paths.
• Quantized with the thinking-capable chat template intact; evaluation used enable_thinking=false for MMLU-Pro/GSM8K and thinking-on sampling for AIME/GPQA/LCB.

Paper: Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM (arXiv:2609.04098).