Qwen3.6-27B-NVFP4

Qwen3.6-27B — NVFP4 (mixed precision, compressed-tensors)

An NVFP4 (4-bit floating-point, NVIDIA Blackwell) quantization of Qwen/Qwen3.6-27B, produced with llm-compressor and shipped in compressed-tensors format for vLLM.

The recipe deliberately mirrors NVIDIA's official nvidia/Qwen3.6-27B-NVFP4 so the two are directly comparable. Qwen3.6-27B is a hybrid architecture (48 gated-delta linear-attention/SSM layers + 16 full-attention layers, dense MLP, plus a vision tower and an MTP head). FP4 is applied only to the large feed-forward matmuls; the sensitive recurrent/attention path is kept at FP8; the SSM control projections, norms, embeddings, MTP head and vision tower stay in bf16.

Precision map

Calibration: 512 samples (~755k tokens, ~30% reasoning traces), seq len 2048. Calibration data is used only for the FP8-static activation scales and the FP8 KV-cache scales. The NVFP4 weight scales are computed from the weights (RTN) and do not use calibration — this matches NVIDIA.

Deviations from nvidia/Qwen3.6-27B-NVFP4 (stated explicitly)

1. lm_head is bf16 here; NVIDIA quantized it to NVFP4. vLLM's compressed-tensors loader for this architecture does not support a quantized lm_head (ParallelLMHead exposes only lm_head.weight), so a quantized lm_head fails to load. This is a forced, format-level deviation — not a tuning choice. Consequence: this checkpoint is ~0.7 GB larger than NVIDIA's, and keeps lm_head at higher precision. Any accuracy difference attributable to lm_head is a memory-budget artifact, not a quality win.
2. Format: this checkpoint is compressed-tensors; NVIDIA's is modelopt. Same underlying NVFP4/FP8 numerics; different loader path in vLLM.

Everything else (which layers are NVFP4 vs FP8 vs bf16, group size 16, FP8-static activations, FP8 KV cache) matches NVIDIA.

Usage (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(
    model="IKilledBardia/Qwen3.6-27B-NVFP4",
    trust_remote_code=True,
    kv_cache_dtype="fp8",              # use the calibrated FP8 KV cache
    kernel_config={"enable_flashinfer_autotune": False},
)
print(llm.generate(["<|im_start|>user\nWhat is 84*3/2?<|im_end|>\n<|im_start|>assistant\n"],
                   SamplingParams(temperature=0, max_tokens=64))[0].outputs[0].text)

Requires a Blackwell GPU (B200 tested). NVFP4 W4A16 runs weight-only via the Marlin kernel — memory savings, not full FP4-tensor-core compute.

Evaluation

<!--EVAL_TABLE-->

Measured with vLLM. bf16 baseline uses bf16 KV cache; both NVFP4 models use FP8 KV cache. gsm8k is 0-shot with the chat template and a 4096-token generation budget (the model reasons in <think> blocks); wikitext perplexity and MMLU are computed via lm-evaluation-harness (loglikelihood).

Reproduction

Quantized with llm-compressor==0.12.0, compressed-tensors==0.17.0, transformers==5.10.1; served/evaluated with vllm==0.24.0.