Qwen3.8-27B-FP8

Qwen3.8-27B-FP8

FP8_DYNAMIC W8A8, attnbf16 — FP8 (E4M3) weights with dynamic per-token activation quantisation, applied to the MLPs only. The entire self_attn path, the linear_attn (SSM) block, the vision tower, lm_head and the MTP head all stay in bf16.

A quantisation of Qwen/Qwen3.8-27B — a dense, multimodal Qwen3.5-family model (qwen3_5) with hybrid linear + full attention, a vision tower, and a multi-token-prediction head.

Why only the MLPs

Qwen3.8-27B sets attn_output_gate: true, which fuses the attention output gate into q_proj — making it [12288, 5120] instead of [6144, 5120]. Half of that tensor is a multiplicative per-head gate on what attention writes into the residual stream, and quantisation error on a multiplicative gate behaves worse than on an additive projection.

Only 16 of the 64 layers are full_attention (the rest are linear_attn), so the whole self_attn block is a small share of the parameters — holding it in bf16 costs little while removing the riskiest quantisation target. The MLPs supply nearly all of the compression.

The linear_attn (Mamba/SSM) block is kept in bf16 for the same reason it is in every build in this line: quantising the recurrent projections measurably degrades this architecture.

Measurements

KLD is per-token, 8 samples from neuralmagic/calibration (LLM split) at max_seq=1024. PPL is wikitext-2-raw test, non-overlapping 2048-token chunks. Baseline for both is bf16 Qwen/Qwen3.8-27B itself, so what is measured is quantisation loss and nothing else.

Two KLD columns, because one number cannot be fair to both checkpoints. The official release is DeepSeek-format FP8, which has to be dequantised to plain bf16 before it will run at all on Blackwell — and a dequantised checkpoint runs with bf16 activations. Our checkpoint, loaded normally, still applies its scheme's dynamic FP8 activation quant. Comparing those two directly measures W8A8 against W8A16 and flatters whichever model was dequantised. So both are reported both ways.

Weight-only = weights round-tripped through the quantised format, activations bf16. As deployed = the same weights plus dynamic per-token FP8 activation quantisation on exactly the modules each checkpoint actually converts. For this build the two differ because FP8_DYNAMIC quantises activations; for the official they differ for the same reason.

This build has lower divergence from the bf16 parent than the official release on both measures — 31% lower weight-only and 24% lower as deployed — while quantising 192 modules to the official's 407.

On PPL the two are not meaningfully separable; both sit within noise of the bf16 parent. A negative ΔPPL does not mean this build beats its own parent, only that the loss is below measurement noise.

What does not explain the difference

The intuitive story is that scale layout matters — the official uses 128x128 block scales, llm-compressor emits one scale per output channel. We tested that directly, simulating the FP8 E4M3 round-trip on real weight tensors from this model (experiment_block_scales.py):

All FP8 variants land within 0.1 % of each other — including per-tensor, with a single scale for the entire matrix. The reason is that FP8 is a floating-point format: every element carries its own exponent, so the scale only has to bring values into range rather than resolve dynamic range the way a fixed-point scale must. The INT8 control shows the method is sensitive enough to detect layout effects where they exist (a 7.4x spread) — and that in that regime per-output-channel actually beats 128x128 block, having more scales.

Block scaling does buy a ~5x lower worst-case weight error; that is an outlier-tail property, not an average one.

We also checked whether the vendor used a smarter rounding than round-to-nearest, by comparing their stored weights against an RTN round-trip of the same parent tensors (experiment_rounding.py): their reconstruction error matches RTN to within 0.3 %. They used plain round-to-nearest.

So the gap is not scale layout and not rounding — it is which modules get quantised. This build leaves the attention path, the SSM block and the MTP head in bf16; the official quantises all of them.

So the reasons to prefer this build are architectural, not average-case accuracy:

• the SSM and attention paths are untouched bf16, which is what you want if you care about long-context or multi-turn behaviour and are unwilling to accept quantised recurrent projections;
• it is compressed-tensors format, so it loads in transformers on Blackwell (SM120). The official checkpoint is DeepSeek-format FP8 (quant_method: fp8 + weight_scale_inv, block-128), which does not — transformers' finegrained_fp8 integration has no SM120 recipe, and it has to be hand-dequantised even to be evaluated. (That dequantisation is how the official row above was measured, on this same harness, so the comparison is apples-to-apples.)

Both FP8 rows measure weight precision loss only; neither models dynamic activation quantisation, so the caveat applies equally to both and does not favour either.

MTP / speculative decoding

All 15 mtp.* tensors are present in bf16, re-grafted from the bf16 parent after quantisation (llm-compressor does not save modules transformers never instantiates).

The 8 MTP Linear modules are listed in quantization_config.ignore. This matters: without it, vLLM's compressed-tensors loader treats the bf16 head as a quantised target, finds no scales, mis-loads it, and every draft is rejected — 0 % acceptance at every position while the head still appears to load correctly in the logs.

Measured on vLLM with --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}' over greedy decoding of mixed chat/code/reasoning prompts:

Smoke tests

Text generation, vision, and MTP key presence all pass:

• MTP tensors present: True
• Text: True
• Vision: True — the tower correctly described the test image (The user wants a single, concise sentence describing the image. I need to identify the key subject and setting...)

Inference

vLLM (with MTP speculative decoding)

vllm serve huginnfork/Qwen3.8-27B-FP8 \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --max-num-seqs 32 \
    --quantization compressed-tensors \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

--max-num-seqs 32 is not optional on this architecture: the default (1024) exceeds the available Mamba cache blocks and vLLM aborts CUDA-graph capture. FP8 GEMM runs native on Hopper (SM89+) and Blackwell.

transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

repo = "huginnfork/Qwen3.8-27B-FP8"
proc = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
)

Provenance

Built with llm-compressor 0.13.0 + compressed-tensors 0.18.0 on transformers 5.14.1, torch 2.13.0+cu130, on an RTX PRO 6000 Blackwell (SM120). Plain quantisation — no abliteration, no fine-tuning.

Pipeline, recipes and every measurement JSON (shipped in this repo alongside the weights): https://github.com/huginnfork/llm-quantisation