Qwen3.8-Flash-Next-Uncensored-FP8-lm-head

Qwen3.8-Flash-Next — FP8 output head (lm_head) delta

A drop-in FP8 (E4M3, per-channel) lm_head for lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE (and, by construction, the orcarouter/Qwen3.8-Flash-Next-Uncensored-* family), plus the small vLLM patch that lets vLLM load a quantized ParallelLMHead, the exact quantisation script, and the benchmark scripts + raw results.

Result: lm_head 1.27 GiB -> 0.64 GiB, decode +6–20 % steps/s at one stream, GSM8K identical, MMLU within 1.5 points (n=200 each).

The output head is a dense [248320, 2560] projection read on every target decode step, so halving its bytes is a fixed per-step saving. It also projects the drafter's logits, because this model's MTP head shares the main lm_head.weight.

Files

FP8/lm_head_fp8.safetensors   # lm_head.weight (F8_E4M3 [248320,2560]) + lm_head.weight_scale (BF16 [248320,1])
scripts/quantize_head_fp8.py  # reproduce the FP8 head from the base bf16 head (per-channel, scale = amax/448)
scripts/apply_fp8_head.py     # splice it into a local copy of the checkpoint (updates index + config.json)
vllm/clean-model.py           # patched vllm/models/qwen3_8_flash_next/nvidia/model.py
vllm/clean-mtp.py             # patched .../mtp.py
vllm/Dockerfile.fp8head-v1    # FROM qwen38-blazux-direct:20260909, COPY the two files
bench/                        # bench_ab.py, bench_downstream.py, bench_speed.py, needle.py, raw results

Method

lm_head.weight bf16 -> FP8 E4M3 with a per-output-channel scale (scale = amax / 448, stored BF16), matching the compressed-tensors scheme (naive-quantized, 8-bit, strategy=channel, symmetric) already used by 300 attention/shared layers in this checkpoint. Added lm_head + re:.*lm_head$ to that config group and removed it from ignore.

The head's vLLM module name is language_model.lm_head (not lm_head), which is why the config target is a regex.

vLLM patch (2 files, 3 edits)

1. model.py main head: ParallelLMHead(..., quant_config=without_modelopt_fp4(self.quant_config))
2. mtp.py drafter head: ParallelLMHead(..., quant_config=self.quant_config)
3. a small helper _set_head_linear_attrs(head) that gives the quantized ParallelLMHead the linear attributes the FP8 W8A16 Humming kernel reads (output_partition_sizes, input_size, input_size_per_partition, has_bias, output_size).

Stock vLLM fails on a 4/8-bit ParallelLMHead: the compressed-tensors scheme creates the parameters (and the loader reads lm_head.weight_scale) but process_weights_after_loading then needs those linear attrs.

Capability (n=200 each, enable_thinking=false, temperature 0)

Differences are 0–3 questions, inside the ±2 % sampling band.

Two exact, head-only measurements:

• weight error: relative Frobenius 2.88 %, spectral ‖ΔW‖/‖W‖ = 0.267 %;
• head A/B on 11,476 real final-layer hidden states (6,000 sampled): top-1 agreement 95.12 %, top-5 overlap 0.959, mean KL(bf16‖fp8) 0.0016, and every top-1 flip sat at a near-tie (top1−top2 gap median 0.039 vs 1.733 overall).

Hardware & software (all speed numbers below)

• GPU: 1× NVIDIA DGX Spark (GB10, sm_121), 128 GB unified LPDDR5X (~273 GB/s).
• CPU: 20 cores; serving pinned to the 10 Cortex-X925 cores (cpuset 5-9,15-19).
• Engine: vLLM v0.1.dev20073+g8e685d198, container qwen38-blazux-direct:20260909
  • this repo's FP8-head patch (tag qwen38-fp8head:v1).
• Serving config: --kv-cache-dtype fp8 --async-scheduling --kv-cache-memory-bytes 19327352832 --gpu-memory-utilization 0.85 --max-model-len 500000 --max-num-seqs 8, --speculative-config {"method":"mtp","num_speculative_tokens":2}.

Absolute tok/s are hardware- and content-dependent. The relative saving from halving the head is what transfers to other GB10-class boxes.

Speed

Decode, single stream, real prompts, after warm-up convergence:

Speculative acceptance is unchanged (code 87–88 %, prose 74–77 %, zh 66–70 %).

Measurement protocol and variance (read this). This box is memory-tight and the ~48 GB PLE table is demand-paged, so requests right after an idle period read cold NVMe pages and run 10–30 % slower. Each configuration was therefore measured as 6 identical bench cycles, discarding the first 2 and averaging cycles 3–6; with a cold first cycle the same FP8 config reads 35–44 tok/s instead of ~49. The ~+20 % figure is the converged value, not a best case: the "+8.8 %" in this card's first revision was exactly such an under-warmed single cycle and has been corrected.

Prefill is unaffected — the head is applied to the last token only, so its cost is negligible against the transformer stack: 802 / 1327 / 1608 / 1503 / 1412 tok/s at 1k / 4k / 16k / 64k / 128k tokens (TTFT 1.3 / 3.1 / 10.2 / 43.6 / 92.9 s, FP8 head, distinct prefixes). Needle retrieval at 13k and 53k tokens: correct.

Reproduce

# 1) quantize the head (bf16 -> fp8 e4m3 per-channel)
python scripts/quantize_head_fp8.py --model-dir /path/to/base --out lm_head_fp8.safetensors

# 2) apply to a local copy (writes model-00017, updates index + config.json; backs up first)
python scripts/apply_fp8_head.py --model-dir /path/to/base --head lm_head_fp8.safetensors

# 3) build the patched image
docker build -f vllm/Dockerfile.fp8head-v1 -t qwen38-fp8head:v1 .

# 4) serve (same flags as the base stack)
vllm serve <model-dir> --kv-cache-dtype fp8 --async-scheduling \
  --kv-cache-memory-bytes 19327352832 --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Limitations

• Measured on one DGX Spark (GB10), vLLM v0.1.dev20073+g8e685d198, fp8 KV, MTP=2, --kv-cache-memory-bytes 18 GiB. Absolute tok/s are hardware/content dependent; the relative head saving is what transfers.
• Capability numbers are GSM8K/MMLU subsets (n=200), not full suites.
• A ~5 % argmax flip rate at near-ties is inherent to any head quantisation; downstream quality was unchanged within sampling error here, but this is not a proof of zero degradation on every task.
• The mkldnn-style naive-quantized scheme is per-channel; fine-grained block-128 FP8 was intentionally not used (different quant_method, and mixing it into a compressed-tensors checkpoint is not supported).

Attribution / license

Base weights: lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE, derived from orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 and ultimately Qwen/Qwen3.8-Flash-Next. The Qwen Community License 1.0 and the base model's terms apply to any redistribution of these derivatives. The vLLM patch is Apache-2.0 (vLLM), on top of NVIDIA's container image.