Qwen3.8-27B-MXFP8

Qwen3.8-27B-MXFP8

MXFP8 (OCP microscaling FP8, block size 32) quantization of Qwen/Qwen3.8-27B, produced with NVIDIA TensorRT Model Optimizer 0.45.0.

Checkpoint size is 32.0 GB across 8 shards.

What is quantized

Only the language-model linear layers are converted to MXFP8. 155 modules are left in BF16:

KV cache is not quantized (kv_cache_quant_algo: null).

The built-in MTP head is preserved, so {"method":"mtp"} speculative decoding still works.

Hardware support

MXFP8 has native block-scaled tensor-core support only on Blackwell (SM100+). On older hardware, engine support varies:

• vLLM — runs on SM80+. The Marlin kernel dequantizes weights and computes in BF16, so pre-Blackwell you get the memory footprint of FP8 with BF16 compute, not FP8 math.
• SGLang — refuses to load below SM100 (get_min_capability() returns 100 for MXFP8); the checkpoint will not start on Ada or Hopper without forcing a block-FP8 conversion path.

This checkpoint was verified serving on 2x RTX 6000 Ada (SM 8.9) under vLLM via the Marlin path.

Serving with vLLM

vllm serve chriswritescode/Qwen3.8-27B-MXFP8 \
  --quantization modelopt_mxfp8 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --attention-backend flashinfer \
  --trust-remote-code

Evaluation: KL divergence from BF16

KL(BF16 reference ‖ candidate), full vocabulary, no top-k, over 136 held-out contexts of 2,048 tokens (278,392 scored positions). Hidden states are captured at the final norm and both operands are replayed through one shared BF16 LM head, so no candidate's own head quantization is counted. Confidence intervals are a bootstrap over the 41 source clusters, 10,000 samples.

Paired on identical contexts, this checkpoint is −0.004894 lower, 95 % CI [−0.006547, −0.003522], winning 136 / 136 contexts — 36.6 % less divergence from BF16 than the official FP8 release.

By stratum:

Controls

Both candidates were captured on the same hardware (RTX 6000 Ada, TP1, stock vLLM, enforce_eager, prefix caching off) against the same BF16 reference.

• Replay reproducibility — re-scoring the suite author's own published FP8 captures through this pipeline gives 0.013201 against their published 0.013126 (+0.6 %). Inputs are SHA-256-identical, so that residual is GPU-arithmetic difference, not method difference.
• Capture-environment floor — official FP8 captured here versus the same model captured by the suite author differs by +0.000164, 95 % CI [−0.000096, +0.000434]. The interval includes zero, so this environment introduces no systematic bias. The measured advantage above is 30x that floor.

What this does and does not say

This measures distributional fidelity to BF16, not task accuracy. No public task benchmark has been run on this checkpoint. Absolute KLD magnitudes are suite-specific; the paired comparison against official FP8 on identical contexts is the transferable result.

Corpus, BF16 reference captures, shared LM head and harness come from malaiwah/qwen38-27b-fidelity-suite-v3 and malaiwah/qwen38-27b-exl3 (tools/fidelity.py), used unmodified.

License

Apache 2.0, inherited from the base model. See LICENSE.