Qwen3.8-27B-NVFP4-AWQ-GPTQ

Qwen3.8-27B-NVFP4-AWQ-GPTQ

Mixed-precision NVFP4 quantization of Qwen/Qwen3.8-27B, built with llm-compressor using AWQ activation-aware scaling followed by GPTQ, with an imatrix_mse observer.

24.7 GB. At the same size as a plain NVFP4 build, it cuts high-confidence damage by roughly a third, and it is the most accurate NVFP4 checkpoint of this model we have measured.

Recipe

Two passes, in order:

1. AWQ — per-input-channel scaling on post_attention_layernorm → {gate_proj, up_proj} and up_proj → down_proj. Gate and up share one input, so the reciprocal scale folds into the norm weights: the accuracy gain costs zero bytes and zero throughput. The scales merge into weights entirely, so unlike rotation-based methods (QuIP/SpinQuant) this checkpoint still runs under tensor parallelism.
2. GPTQ on every quantized module (actorder="static", dampening_frac=0.01).

Calibration: 1024 sequences × 1024 tokens of a balanced Nemotron-v2 blend (25% code, 25% math, 20% STEM, 20% chat, 10% multilingual).

lm_head and embed_tokens are left in BF16 — matching Qwen's own official FP8 release, which does the same.

Benchmarks

Measured against the BF16 base model on 142,727 tokens of self-distilled thinking-mode output, plus 200 free greedy generations. vLLM 0.27.1, TP=2, 2×B300.

Bold marks the best value in each column among the FP4 checkpoints; the FP8 row is a reference at a different precision and size class, so it is excluded from the comparison. All sizes are on-disk tensor bytes and include the ~0.85 GB BF16 MTP head, which every checkpoint in this table ships. Subtract ~0.85 GB for a no-MTP comparison.

Columns. top-1 is raw argmax agreement with BF16. The four bucket columns are disagreement rates, split by how confident the base model was at that position (top1−top2 logprob margin): near-tie <0.5, moderate 0.5–2, confident 2–5, certain >5. Only confident and certain represent real damage — a flip where the base model itself was nearly tied is numerical noise, not a quality loss. divmed is the median token index at which free greedy generation first diverges from BF16 (higher is better).

Perplexity is deliberately excluded. On this comparison it is anti-correlated with quality — the checkpoint with the best perplexity (RadixArk, −1.75%) has the worst certain-bucket damage of any arm measured (0.70%, 3.7× this model's). Do not rank FP4 checkpoints of this model by perplexity.

In an internal ablation, removing the AWQ pass and keeping everything else identical raises confident damage from 2.69% to 3.97% — so AWQ closes about half of the gap to FP8, at no size or speed cost.

Usage

from vllm import LLM
llm = LLM("selimaktas/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2)

Requires a Blackwell-class GPU for native NVFP4, and vLLM with compressed-tensors.

Speculative decoding (MTP)

The model's MTP (multi-token prediction) head is included, in BF16, and works with vLLM's mtp speculative decoding:

from vllm import LLM
llm = LLM("TelperionAI/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2,
          speculative_config={"method": "mtp", "num_speculative_tokens": 2})

Qwen3_5ForConditionalGeneration does not carry mtp.* in its state dict, so llm-compressor never sees it and it is silently dropped, even though config.json still declares mtp_num_hidden_layers: 1. It is grafted back in here from the base checkpoint and excluded from quantization (re:.*mtp.* in quantization_config.ignore; without that exclusion the quantization target regexes also match mtp.layers.0.mlp.* and vLLM fails to load). Draft quality drives acceptance rate, so it is kept at full precision rather than quantized.

Acceptance rate has not been measured; the head is verified to load and generate.

Limitations

• Single evaluation corpus. All numbers come from one self-distilled corpus. The margins over the public NVFP4 checkpoints are large and statistically solid, but the comparison has not been replicated on a second distribution.
• Vision tower is untouched (BF16); this was evaluated as a text model.

KV-cache quantization (calibrated scales included)

This checkpoint carries per-layer k_scale / v_scale in model-kv-scales.safetensors, with kv_cache_scheme under quantization_config — the on-disk contract llm-compressor emits and vLLM reads in BaseKVCacheMethod. Without them vLLM falls back to a scale of 1.0.

--kv-cache-dtype fp8      # 32 KiB/token, from 64 KiB at fp16

Provenance: these scales were measured on Qwen3.8-27B-NVFP4-AWQ-AutoRound, a different quantization of the same base model, over a 2,353-document / 3.7M-token corpus — the size at which the post-RoPE V absmax stops moving. K/V range is a property of the model rather than of the weight format (per-layer amax agrees to a median ratio of 1.0002 across our trellis / FP8 / INT4 / NVFP4 builds), so they transfer. They were not measured on this checkpoint specifically; cross-build agreement is excellent at the median but ranges 0.88–1.36 in the tails.