Qwen3.8-27B-fp8-KV-scale-calibration

Qwen3.8-27B fp8 KV-cache scale calibration (vLLM side-car)

32 calibrated k_scale/v_scale scalars for Qwen3.8-27B's 16 full-attention layers (the GDN/linear-attention layers have no paged KV, so they're excluded), packaged so they drop straight into any Qwen3.8-27B hybrid-GDN checkpoint that serves with --kv-cache-dtype fp8 on vLLM.

Why this exists

Every Qwen3.8-27B checkpoint we've inspected that advertises fp8 KV cache in its config.json (RadixArk's NVFP4 export included) ships no k_scale/v_scale tensors at all. vLLM then falls back to an uncalibrated scale of 1.0 and logs a warning_once that's easy to miss: the MTP draft layer has no scales by design, so the warning fires even when every target-model layer loaded correctly — a normal boot log can't tell you whether calibration succeeded.

unsloth/Qwen3.8-27B-NVFP4 is the one exception we found: it ships real static fp8 KV calibration tensors (llm-compressor one-shot, per-tensor symmetric) that its own README never mentions. But there's a second, independent bug: vLLM's Qwen3.5-family loader (qwen3_5.py) never calls maybe_remap_kv_scale_name (unlike gemma4, hy_v3, gpt_oss, which do), so even unsloth's checkpoint-provided scales are silently ignored by every vLLM user running that family — filed as vllm#54623, fix proposed in vllm#54624.

The tensors in model-kvscales.safetensors are unsloth's calibrated values, extracted and renamed to vLLM's internal Attention-module parameter path (...self_attn.attn.{k,v}_scale) rather than the checkpoint convention — that specific name loads unconditionally through the generic weight-name matcher regardless of the qwen3_5.py gap, so it works today on v0.27.1/ v0.28.0 without waiting for #54624 to merge.

Validation

• Cross-checked against a second, independently-calibrated Qwen3.8-27B export (different quant recipe, different calibration run) — matched to 3–4 decimal places on every shared layer (e.g. layer 11 k_scale 1.2549 vs 1.2539, layer 23 k_scale 1.3057 vs 1.3057). Two unrelated calibrations converging this tightly means KV amax is a stable property of the base model, not calibration noise.
• Measured impact on our own serving stack (Qwen3.8-27B-NVFP4, RTX 5090, vLLM v0.27.1, GSM8K greedy, n=300): uncalibrated fp8 KV 92.3%, calibrated (this graft) 93.7%, bf16-KV control 94.5% — recovers roughly two-thirds of the fp8-KV accuracy tax at zero cost in memory or context length.

Use

pip install safetensors torch
hf download joelafrite/Qwen3.8-27B-fp8-KV-scale-calibration --local-dir kv-scales && cd kv-scales
python apply_kv_scales.py --target /path/to/your/Qwen3.8-27B/checkpoint --dry-run   # inspect first
python apply_kv_scales.py --target /path/to/your/Qwen3.8-27B/checkpoint             # writes one new shard + updates the index

Only adds a new shard and updates model.safetensors.index.json (.bak backup kept); never touches your existing weight files. Works regardless of the weight quantization scheme (NVFP4, W4A16-GPTQ, AWQ, …) — this calibrates the KV cache activation path, which is orthogonal to how the linear layers are quantized. If your export doesn't use vLLM's default layer-count/indexing (some layers pruned, different attention-layer positions, etc.) run --dry-run first and check the layer indices line up.

To confirm it actually loaded (the boot-log warning can't tell you, see above): boot once with VLLM_LOGGING_LEVEL=DEBUG and grep for Loaded weight ... self_attn.attn.k_scale — expect 16 hits.

Credit

Calibration values sourced from unsloth/Qwen3.8-27B-NVFP4's own shipped k_scale/v_scale tensors — full credit to unsloth for running that calibration; this repo just extracts, renames, and packages it as a generic side-car so it's usable on other checkpoints of the same base model, plus documents the two upstream vLLM gaps that make it silently useless without the rename.

License

MIT (script + packaging). The calibration values themselves are 32 floating-point scalars derived from unsloth's public checkpoint.