Qwen3.8-27B-Opus-Distill-v2-FP8

Qwen3.8-27B-Opus-Distill-v2-FP8

Block-wise FP8 (e4m3, 128x128 blocks, dynamic activation scaling) quantization of barozp/Qwen3.8-27B-Opus-Distill-v2, built with the exact recipe Qwen/Qwen3.8-27B-FP8 uses for the base model -- read live from that repo's quantization_config at build time, not guessed. No calibration dataset is needed (dynamic activation scheme): only weights are quantized, block-wise, from their own value range. Roughly 56% of the source bf16 size (~31 GB vs ~55 GB).

Excluded from FP8 (kept in original precision, same policy as Qwen's own FP8 release): vision tower, MTP head, norm/gate layers, embed_tokens, lm_head, and the Gated-DeltaNet SSM-specific parameters (A_log, conv1d, dt_bias, internal norm). Everything else (attention QKVO on full-attention layers, FFN, Gated-DeltaNet's own in_proj/out_proj matrices) is quantized.

Changelog

• 2026-08-21 -- IMPORTANT config fix (revision 67ce584). If you pulled this repo before that date and sglang fails to start with ValueError: Weight output_partition_size = 48 is not divisible by weight quantization block_n = 128, delete your local copy / re-download so you get the fixed config.json. Root cause: the modules_to_not_convert list must use bare module paths (...linear_attn.in_proj_ba) -- sglang resolves fused GDN layers through their shard names (in_proj_b, in_proj_a), so tensor-name-style entries (....weight) never match and a merged projection got FP8'd despite being bf16 in the checkpoint. Weights were always correct and are unchanged.
• 2026-08-22: model card rewritten with native-FP8 serving verification and serving guide.

Validation status

Known environment limitation (not specific to this checkpoint): flashinfer's JIT arch-check currently rejects consumer Blackwell (sm_120) and bites in up to three places: attention planning during CUDA-graph capture, top-k/top-p sampling at request time, and EAGLE verify sampling during speculative decoding. Workarounds are documented below. One consequence on sm_120 specifically: a fully non-greedy MTP-accelerated user request cannot be exercised there today (the EAGLE verify sampler hard-wires flashinfer via sgl_kernel); greedy warmup requests do pass. On Ada/Hopper (sm_89/sm_90) flashinfer works normally and none of these paths are affected. vLLM could not be independently verified in our test image for dependency-conflict reasons unrelated to this checkpoint; the format matches the standard HF FP8 layout used by Qwen's own releases.

Paired BF16-vs-FP8 benchmarks

Both checkpoints were evaluated under an identical protocol so the delta isolates true quantization cost from any serving-stack numerics:

Every delta sits inside one standard error (~±0.02): the FP8 build's quality cost on these tasks is statistically indistinguishable from zero. Sanity check: our BF16 rerun reproduced this card's published reference values almost exactly (0.8476 / 0.7500 / 0.6220), confirming both the card and the protocol.

Test environment (reproduction)

• Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition (sm_120), 97 GB
• Software: lm-evaluation-harness 0.4.12 · transformers 5.15.0 · torch 2.11.0+cu128
• Backend: lm-eval HF backend (--model hf) — no serving layer, no batching nondeterminism
• dtype bfloat16 for both checkpoints · chat template OFF · 0-shot (--num_fewshot 0)
• --limit 500 (QUICK mode) · --batch_size 16 · --seed 1234

Command shape (run once per checkpoint):

python3 -m lm_eval \
    --model hf \
    --model_args pretrained=<CHECKPOINT>,dtype=bfloat16,trust_remote_code=True \
    --tasks mmlu,hellaswag,arc_challenge \
    --num_fewshot 0 \
    --batch_size 16 \
    --limit 500 \
    --seed 1234 \
    --output_path <OUT_DIR>

Notes:

• The FP8 checkpoint executes through native FP8 matmul in transformers 5.x, which needs pip install "kernels==0.16.0" — otherwise you'll hit ImportError: finegrained-fp8 kernel unavailable at the first FP8 linear layer.
• Wikitext perplexity is excluded from this pairing: its rolling-loglikelihood evaluation triggers O(seq²) decay-mask allocation in this architecture's pure-torch fallback path and OOMs even on 97 GB GPUs.

Serving with sglang

Standard launch on Ada/Hopper (sm_89 / sm_90 -- e.g. RTX 4090, RTX 6000 Ada, H100):

python3 -m sglang.launch_server \
    --model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \
    --trust-remote-code

Consumer Blackwell cards (sm_120: RTX PRO 6000, RTX 5090) currently need three workaround flags until flashinfer fixes its sm_120 detection:

python3 -m sglang.launch_server \
    --model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \
    --trust-remote-code \
    --attention-backend triton \
    --disable-cuda-graph \
    --sampling-backend pytorch

Optional MTP / speculative decoding (NEXTN) -- add to either command above:

    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

(If you are on sm_120, include the three workaround flags here too.)

Quick start (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "barozp/Qwen3.8-27B-Opus-Distill-v2-FP8", dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill-v2-FP8")

On GPUs below compute capability 8.9, transformers will automatically dequantize to bf16 at load time (you'll see a warning) -- this still works correctly, it just won't give you the FP8 memory/speed benefit.

Related

• barozp/Qwen3.8-27B-Opus-Distill-v2 -- source model, full card.
• barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF -- GGUF quantizations (BF16 down to IQ1_M) for llama.cpp.
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit and MLX-8bit -- Apple Silicon MLX conversions via mlx-vlm.
• Qwen/Qwen3.8-27B-FP8 -- the base model's official FP8 release, whose recipe this follows.