Qwen3.6-27B-FP8-lmhead-fp8

Qwen3.6-27B-FP8 with FP8-quantized lm_head (vLLM PR repro artifact)

🚀 RTX 5090 update: we got the official FP8-weight path running on a single 32 GB RTX 5090 by combining vLLM hybrid TurboQuant with FP8 quantization of the large lm_head tensor.

This checkpoint is the lm_head-FP8 half of that experiment. It is loadable today with the C4 overlay image documented below. Stock vLLM ≤ 0.20 cannot load it without source patches; once vllm-project/vllm#41000 merges and lands in a release image, vLLM will load it as-is.

Quality has not been re-validated under the final builder-built C4 image yet. Treat this as an RTX 5090 FP8 deployment candidate / PR repro artifact, not as a general production recommendation.

ℹ️ Not recommended: the NVFP4 alternative. There is also an inferRouter/Qwen3.6-27B-NVFP4 build that fits a single RTX 5090 32 GB, but its quality is materially worse than the upstream FP8 — same regression we observed across every NVFP4 27B variant we evaluated. The C4 path here (official FP8 weights + hybrid TurboQuant + FP8 lm_head) is the route that preserves upstream FP8 quality on the 32 GB card.

Summary

Derivative of Qwen/Qwen3.6-27B-FP8 with one specific change: the lm_head projection has been block-FP8-quantized ([128, 128] blocks, e4m3fn, BF16 scales, DeepSeek-V3-style weight_scale_inv) to free up GPU memory on tight cards (RTX 5090 32 GB).

Net VRAM saving on GPU at load time: ~1.18 GiB (BF16 → FP8 of one tensor).

This matches what is observed on the C4 stack: BF16-lm_head model load reads as 27.66–27.69 GiB; FP8-lm_head model load reads as 26.5 GiB on RTX 5090. The earlier "~0.32 GiB real saving" measurement (vLLM 0.19.1 + RTX 5090) was a Triton-autotune-budget artefact — autotune scratch was absorbing most of the weight-side delta. Under vLLM 0.20 + the hybrid TurboQuant runtime (PR #39931) the autotune scratch is paid out of a different bucket and the full FP8 delta falls through to the "Model loading took" line.

All other layers and config are byte-identical to the upstream Qwen/Qwen3.6-27B-FP8 checkpoint. The only modified files are:

• outside.safetensors — lm_head.weight re-quantized + lm_head.weight_scale_inv added
• model.safetensors.index.json — added lm_head.weight_scale_inv entry
• config.json — quantization_config.modules_to_not_convert no longer lists lm_head; quantization_config.lm_head: true added as opt-in flag

Why this exists

Qwen3.6-27B-FP8 BF16-lm_head weights load as ~27.66 GiB on a 32 GB GPU, leaving only ~1 GiB headroom. That is not enough for Triton autotuner scratch (GDN/Mamba solve_tril kernels) + KV cache blocks, so the engine OOMs at startup or during inference even with conservative gpu_memory_utilization and max_model_len.

Quantizing lm_head (the largest BF16 tensor remaining outside the FP8 quant scope of the upstream variant) frees ~1.18 GiB. Combined with hybrid TurboQuant KV (PR #39931), this is enough to pass the previously failing profiling/startup path in the C4 experiments on RTX 5090 (which previously failed with "Tried to allocate 272 MiB"). The recommended production envelope remains max_num_seqs=3 until the full builder-built image gate and stability test are recorded.

Loadable today

This checkpoint loads end-to-end on RTX 5090 32 GB using the InferRouter C4 overlay image, validated 2026-04-29:

inferrouter/vllm-openai-v020-pr39931-lmheadfp8-turboquant:latest
base:    vllm/vllm-openai:v0.20.0
overlay: PR #39931 (hybrid TurboQuant on Mamba/GDN models)
         + PR #41000 (FP8 ParallelLMHead opt-in) ported in-image
opt-in:  ENABLE_LM_HEAD_FP8=1 in c4_prepare_vllm020_pr39931_overlay.sh

Build script (InferRouter quantization workspace): qwen-36-27b/c4_turboquant/c4_prepare_vllm020_pr39931_overlay.sh.

Single-GPU TP=1 only. Multi-GPU TP>1 raises NotImplementedError from the loader shim — the proper vocab-shard-aware FP8 scale loader is a follow-up tracked behind PR #41000.

Once PR #41000 merges into vLLM main and lands in a release image, the overlay step becomes unnecessary; this checkpoint will load on stock vLLM directly.

Recommended runtime profile (RTX 5090, 32 GB)

--language-model-only
--quantization fp8
--kv-cache-dtype turboquant_k8v4
--gpu-memory-utilization 0.96
--max-model-len 5120
--max-num-seqs 3
--max-num-batched-tokens 15360
--enforce-eager
--no-enable-chunked-prefill
--reasoning-parser qwen3

Conservative fallback if scratch headroom matters more than concurrency: turboquant_k8v4 / max_model_len=4096 / max_num_seqs=3. Optional long-context profile with more KV-quality risk: turboquant_4bit_nc / max_model_len=6144 / max_num_seqs=2. Do not pair turboquant_4bit_nc with max_num_seqs=4 for near-full-context traffic; measured 4352 / 4608 × 4 profiles failed warmup OOM despite the better theoretical KV compression.

Status — why stock vLLM ≤ 0.20 doesn't load this

Three layered gaps in stock vLLM prevent loading this checkpoint out of the box:

1. Fp8Config.get_quant_method does not opt into ParallelLMHead. Returns None for any non-LinearBase/FusedMoE/Attention layer → falls through to UnquantizedEmbeddingMethod, which only registers weight (no weight_scale_inv).

2. Qwen3.5/3.6 model classes (vllm/model_executor/models/qwen3_5.py) build ParallelLMHead(...) without quant_config=, so the dispatcher above is never even called for lm_head.

3. VocabParallelEmbedding.weight_loader asserts loaded_weight.shape[output_dim] == self.org_vocab_size for every parameter routed through it. Companion FP8 params like weight_scale_inv (shape [1940, 40]) trip this assertion.

Tracking issue: vllm-project/vllm#40999 — "[Feature][FP8] Opt-in ParallelLMHead quantization in legacy Fp8Config". Filed 2026-04-27.

PR: vllm-project/vllm#41000 — "[FP8] Add opt-in ParallelLMHead dispatch to Fp8Config". Open since 2026-04-27. Implements all three gaps (Gap 1: dispatcher; Gap 2: qwen3_5 quant_config plumbing; Gap 3: companion-param loader). Mechanically and functionally validated end-to-end on RTX 6000 Pro 96 GB (token emission, deterministic + Czech sanity) and on RTX 5090 32 GB via the C4 overlay (model load + math + Czech sanity).

Related prior art: vllm-project/vllm#35696 ("[Model] Optional FP8 lm_head compression for Llama and Mistral") — different shape (per-model post-load dtype cast, env-var gated), stalled on maintainer review since 2026-03.

Reproducing the failure on stock vLLM

# Pull this checkpoint
huggingface-cli download inferrouter/Qwen3.6-27B-FP8-lmhead-fp8 \
  --local-dir ./qwen36-27b-lmhead-fp8

# Try to serve with stock vLLM (≤ 0.20) — will crash at load_weights:
vllm serve ./qwen36-27b-lmhead-fp8 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --enforce-eager \
  --kv-cache-dtype fp8_e4m3

Expected error:

ValueError: There is no module or parameter named 'lm_head.weight_scale_inv'
in Qwen3_5ForCausalLM. The available parameters belonging to lm_head
(ParallelLMHead) are: {'lm_head.weight'}

(after applying patches #1 + #2 from PR #41000:)

File "vllm/model_executor/layers/vocab_parallel_embedding.py", line 463
    assert loaded_weight.shape[output_dim] == self.org_vocab_size
AssertionError

(with the full PR #41000 applied, or the C4 overlay image, the load succeeds and the engine reaches Application startup complete.)

How the lm_head was quantized

DeepSeek-V3 block-FP8 scheme, matching the existing weight format used by the upstream FP8 layers in this checkpoint:

BLOCK = 128
FP8_E4M3_MAX = 448.0

# For each [128, 128] block of the [248320, 5120] weight:
scale_inv = max(abs(block)) / FP8_E4M3_MAX     # per-block scale, BF16
quantized = clamp(block / scale_inv, -FP8_E4M3_MAX, FP8_E4M3_MAX).to(F8_E4M3)

No calibration data is needed (FP8 dynamic activation scheme). Both dimensions of lm_head.weight are exact multiples of 128 — no padding.

Reproducer script: quantize_lm_head.py in the InferRouter quantization workspace (linked from PR #41000).

License

Inherited from upstream Qwen license (Apache-2.0). All weights are mathematically derived from the upstream checkpoint by deterministic quantization; no retraining or fine-tuning was performed.

Acknowledgements

Built while debugging vLLM FP8 deployment for InferRouter on RTX 5090 32 GB hardware. The three-layer finding above came out of that work and is the basis for PR #41000.