Qwen3.8-27B-FP8-KVcal

Qwen3.8-27B-FP8-KVcal

Official Qwen3.8-27B block-FP8 weights with calibrated FP8 KV-cache scales — a drop-in replacement for Qwen/Qwen3.8-27B-FP8 that serves a properly scaled fp8 KV cache in vLLM.

The only difference from the official checkpoint: one small extra shard (kv_cache_scales.safetensors, 7 KB) with 64 float32 scalars, and the matching entries in model.safetensors.index.json. Every weight byte is byte-identical to the official release (Apache-2.0).

What it fixes

Qwen/Qwen3.8-27B-FP8 ships no KV-cache calibration tensors. With --kv-cache-dtype fp8, vLLM defaults all KV scales to 1.0 and logs kv_cache.py "uncalibrated scaling factor 1.0" warnings — a miscalibrated per-tensor FP8 quantization that assumes K/V activations sit around scale 1.0. On this hybrid (Gated DeltaNet) model the risk is lower than on dense-attention models — K/Q are RMS-normed before hitting the cache, and only 16 of 64 layers are full attention — but the scales are still worth having.

This checkpoint provides, for the 16 full-attention layers (indices 3, 7, 11, ..., 63):

The scales are named along vLLM's runtime internal parameter path (...self_attn.attn.*_scale, not the HF checkpoint convention) so they load unconditionally through the generic weight-name matcher — this works around vllm#54623, where the qwen3_5-family loader never calls maybe_remap_kv_scale_name and silently ignores checkpoint-convention KV scales (fix proposed in vllm#54624). If you maintain a fork/loader, the proper fix is upstream remap support.

Serve it

vllm serve stoyani/Qwen3.8-27B-FP8-KVcal \
  --kv-cache-dtype fp8 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.94 \
  --enable-chunked-prefill \
  --enable-prefix-caching

Verify at boot: the journal should show all 64 KV scales loaded and no "uncalibrated" scale warnings (kv_cache.py:134/148 "scaling factor 1.0" for K/V, :187 "uncalibrated q_scale / prob_scale"). Verified on the stack this was calibrated for: vLLM 0.23.0, NVIDIA RTX PRO 6000 Blackwell (SM120), CUDA 13.

Expected gains on that hardware (measured, not guaranteed elsewhere): KV pool ~1.95x larger (921K → 1,798K tokens at 0.94 utilization), ~2x residency for long-context concurrent workloads; single-stream tok/s roughly flat (compute-bound regime). FP8 KV does not extend the max context length — the Gated DeltaNet state pool is the memory floor on this hybrid.

Honesty notes

• Calibration corpus. Scales were calibrated on 16 documents / 1,100,804 tokens spanning finance, coding, and general text, including two near-full-context documents (262,126 and 262,127 tokens). The corpus is not public (it contains private work documents); the per-layer audit artifacts are in calibration/ (amax + scale per layer, 275 observation calls per layer, status: complete).
• Observation point (and why it matters). The scale must match the distribution that is actually quantized into the FP8 cache. In this model K and Q go through per-head RMS norm (k_norm, q_norm, kept BF16) before the cache write; V is a raw projection output — the architecture has no v_norm. This checkpoint's scales are measured at that cache-write point (post-norm K/Q, raw V). Why the distinction matters: e4m3 is a floating format with 4 mantissa bits, so a wrong-but-bounded scale does not produce garbage — it mis-allocates resolution, and the 448-overflow protection only works when scale = amax/448. Independent calibrations that measure pre-norm activations (e.g. scales extracted from unsloth/Qwen3.8-27B-NVFP4, as packaged in joelafrite/Qwen3.8-27B-fp8-KV-scale-calibration) come out a few dozen times larger — pre-norm K sits around 546–596 while post-norm K is 11.6–23.1. That's likely why such numbers still measure acceptably (relative precision forgives a loose scale to a degree), but the post-norm value is the one that is tight at the observed maximum and provably matches what the kernel receives on the verified stack. V's range (14.9–114.6, growing with layer depth) is wider than K's because V is not normed — expected, not a bug.
• Q/prob scales are dead code today. FP8 attention (quantizing Q and P inside the kernel) is a separate feature from the FP8 KV cache. On the tested stack (vLLM 0.23, SM120, FlashInfer) the quantized-query path is gated off, so q_scale is never applied and prob_scale is float32(1/448). They are included so the :187 warning is gone and the checkpoint is future-proof if a vLLM update enables quantized-query FP8 attention on more hardware. They do not touch K/V handling.
• Scope: per-tensor fp8 only. These scales apply to --kv-cache-dtype fp8 (per-tensor fp8_e4m3). Newer vLLM (0.25.0+) also offers per-(token, head) dtypes (fp8_per_token_head, int8_per_token_head, int4_per_token_head), which compute scales in-kernel at cache-write time and ignore checkpoint scales by design — with those dtypes this sidecar is inert (that path is its own option; on our stack it forced the TRITON_ATTN backend).
• Version sensitivity. Loader behavior around KV scale names changes between vLLM releases. This checkpoint is verified on vLLM 0.23.0 (checked against 0.24.0–0.29.0 notes: per-tensor scale loading unchanged, the #54623 loader gap still open, so the runtime-path naming remains required and conflict-safe). Re-check the boot journal (all 64 scales loaded, no warnings) before trusting it on a newer vLLM.
• Why a pre-calibrated checkpoint is the standard path on 0.28+. vLLM 0.28.0 removed the --calculate-kv-scales CLI option entirely (#49389); on 0.28+ a checkpoint carrying the scales is the only way to get calibrated per-tensor FP8 KV (short of the per-token-head dtypes above).

Benchmark

GSM8K (1,319 test problems, zero-shot, greedy temperature=0, the classic Q: ... A: prompt, no thinking tokens, max_tokens=512), served on NVIDIA RTX PRO 6000 Blackwell (SM120) with vLLM 0.23.0:

Paired results: the calibrated checkpoint is at-or-above BF16 (net +17, 32 FP8-only vs 15 BF16-only correct), and uncalibrated FP8 is within run noise of BF16 (net +7). GSM8K is a short-context benchmark — K/V amax on this model sits well under e4m3's 448 ceiling, so scale 1.0 does not overflow, it just under-uses the format's dynamic range, and e4m3's floating precision forgives that at short context.

Two reading notes:

1. Backend differs by hardware constraint, not by choice. On this SM120 card the installed FLASH_ATTN rejects FP8 KV, so the FP8 run is served by FLASHINFER while the BF16 control picks FLASH_ATTN. The accuracy comparison isolates the KV cache dtype (the backends compute the same math); do not read their throughput difference as a KV-dtype effect. On Hopper-class hardware both would use the same backend.
2. Run-to-run noise. Greedy serving under vLLM's continuous batching is not bit-reproducible across batch configurations (floating-point reduction order); we observed a 3-problem spread between two runs of the exact FP8 set (845 vs 848). Read FP8-vs-BF16 deltas smaller than ~±0.3pp as within run noise.

RULER long-context (128K)

RULER (NVIDIA) — synthetic long-context benchmark. Prompts generated with the official NVIDIA/RULER generator using this model's own tokenizer (seed 42), n=50 per task at 128K, greedy, max_tokens=128, concurrency 4 (identical prompts and settings for all three configs):

Reading:

• Single-needle retrieval at 128K is flawless under all three KV configs — FP8 KV does not degrade long-range retrieval on this model.
• Variable tracking is model-bound, not KV-bound: the BF16 control itself scores only 61% (20/50 samples all configs solve, 8/50 none do). The pairwise deltas (calibrated 33 vs uncalibrated 31 vs BF16 30) favor the calibrated config in every measurement on this and the GSM8K set, but are inside the n=50 noise band — treat as directionally consistent, not a significant difference.
• Why uncalibrated is only mildly worse and not broken: e4m3 is a floating format — relative precision is preserved at any magnitude below the 448 ceiling, so a scale-1.0 miscalibration does not corrupt the cache as long as activations stay bounded. On this model they do (post-norm K/V amax ≤ ~115 observed). The calibrated scale's role is overflow headroom, tight range usage, and correct behavior if your workload's activations exceed the calibration range — the failure mode is saturation at 448, not noise.

Net claim: no measurable long-context degradation from FP8 KV on this stack; the calibrated checkpoint is at-or-above the BF16 control in every measurement, and its practical value is the ≈2× KV pool (residency and long-context decode), not an accuracy gain.

Reproduce:

# 1. generate prompts (official RULER generator, model's tokenizer)
python /path/to/RULER/scripts/data/prepare.py --save_dir ./ruler \
  --benchmark synthetic --task niah_single_1 \
  --tokenizer_path <this-repo> --tokenizer_type hf \
  --max_seq_length 131100 --model_template_type base --num_samples 50 --random_seed 42
# (repeat for --task vt)
# 2. score against a running server
python scripts/eval_ruler.py --data-dir ./ruler --out-dir ./ruler_results --concurrency 4

Reproduce: GSM8K

scripts/eval_gsm8k.py is self-contained (requests + pyarrow, downloads the public parquet from the Hub):

python scripts/eval_gsm8k.py --out-dir gsm8k_results/ \
  --url http://localhost:8000/v1/completions \
  --model stoyani/Qwen3.8-27B-FP8-KVcal

Rebuild / reproduce

The 64 scales are reproducible with the scripts in scripts/:

1. calibrate_kv_scales.py — loads the official checkpoint in-process (vLLM V1, enforce_eager), hooks the 16 full-attention layers to record the running |K|/|V| amax that vLLM actually writes to the FP8 cache, runs your calibration corpus (default: 262,144-token context, 4,096-token prefill chunks), and writes kv_cache_scales.safetensors + calibration_report.json. Scale rule: ceil_to_bfloat16(amax / 448), stored float32 (the smallest BF16-representable scale that preserves the observed range). The script refuses input clipping and incomplete observation, and a failed run records status: failed which the injector rejects.
2. inject_kv_scales.py — copies the official checkpoint (dereferencing LFS symlinks), adds the scale shard, and registers it in model.safetensors.index.json. Requires a complete matching report and 32 finite positive float32 scalars before writing; never modifies weight shards.
3. add_q_prob_scales.py + run_q_prob_extend.sh — optional follow-up that merges 16 calibrated q_scale + 16 prob_scale into the deployed shard in place, keeping the 32 k/v scales bit-identical (a re-run's k/v output is discarded as calibration noise). File-only; backs up shard + index first and self-checks k/v byte-identity after writing.

Files

• 75 files: byte-identical to Qwen/Qwen3.8-27B-FP8 (Apache-2.0) — SHA256SUMS lists them all (the official repo's own crc32.txt was found to be internally stale, so we publish a fresh SHA-256 manifest instead)
• model.safetensors.index.json: official index + 64 scale entries
• kv_cache_scales.safetensors: the 64 calibrated scales (the new content)
• calibration/: per-layer audit artifacts (amax, scale, observation counts)
• scripts/: the calibration, injection, extend, and eval toolchain
• LICENSE: Apache-2.0, matching the upstream weights