Qwen3.6-27B-FP8-dynamic

Qwen3.6-27B-FP8-dynamic

FP8-dynamic (W8A8) quantization of Qwen/Qwen3.6-27B, produced with llm-compressor and served with vLLM.

Model overview

• Base model: Qwen/Qwen3.6-27B (Qwen3.5-family hybrid-attention VLM)
• Quantization scheme: FP8_DYNAMIC (weights quantized to FP8 (E4M3), per-channel, round-to-nearest; activations quantized to FP8 per-token with dynamic scales computed at runtime, no calibration data required).
• What's quantized: the text transformer's Linear layers. The vision tower (re:.*visual.*), the hybrid linear-attention layers (re:.*linear_attn.*), the MoE router (re:.*mlp.gate), and lm_head stay in the original precision.
• Format: compressed-tensors. The full VLM (with vision_config) is saved, so it loads directly in vLLM.
• Memory: ~2x smaller than the bf16 base for the quantized weights.

Deployment (vLLM)

vllm serve soyrsoyr/Qwen3.6-27B-FP8-dynamic

from vllm import LLM, SamplingParams

llm = LLM(model="soyrsoyr/Qwen3.6-27B-FP8-dynamic")
out = llm.generate(["The capital of France is"], SamplingParams(max_tokens=32))
print(out[0].outputs[0].text)

Creation

Quantized with llm-compressor. Because current vLLM only registers the multimodal Qwen3_5ForConditionalGeneration architecture, the model is loaded with AutoModelForImageTextToText and the full model is saved (text quantized, vision tower kept in bf16).

from transformers import AutoModelForImageTextToText, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3.6-27B"
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*", "re:.*mlp.gate$"],
)
oneshot(model=model, recipe=recipe)  # data-free
model.save_pretrained("Qwen3.6-27B-FP8-dynamic", save_compressed=True, save_original_format=False)
tokenizer.save_pretrained("Qwen3.6-27B-FP8-dynamic")

Evaluation

Evaluated with lm-evaluation-harness (OpenLLM Leaderboard v1) and lighteval (generative reasoning suite), both via the vLLM backend.

OpenLLM Leaderboard v1

FP8-dynamic recovers 100.9% of the base model's average score on these tasks, i.e. quantization is effectively lossless here.

MMLU and GSM8k are omitted: Qwen3.6-27B is a reasoning ("thinking") model, and under the OpenLLM v1 protocol both collapse to a harness artifact rather than a real score. GSM8k is generative and its chain-of-thought is truncated by the default max_gen_toks before the final answer, and MMLU's letter-answer loglikelihood is measured at the position where the model wants to emit its <think> block, so every subject lands at random chance (MMLU scored 23.02 for the base model and 23.05 for FP8, both at the 25% four-way chance level, which confirms it is a measurement artifact and not quantization damage). Math and knowledge are better captured by the generative reasoning suite (AIME, MATH-500, GPQA).

Reasoning suite (generative)

The eval that actually reflects a thinking model: single-sample pass@1, model allowed to reason (temperature 0.6, top_p 0.95, up to 32k generated tokens), scored with lighteval.

AIME-25 and MATH-500 are effectively lossless (100.0% and 99.5%). The AIME-24 number is a single-sample (pass@1, n=1) score over only 30 problems, so its standard error is large (~7 points, i.e. roughly a 2-problem swing); the 10-point gap there is within that sampling noise rather than clear quantization damage. For a lower-variance figure one would average over multiple samples (e.g. avg@64), which was out of scope for this single-GPU run. GPQA-diamond will be included once dataset access is granted (it is a gated dataset on the Hub).

Reproduction

OpenLLM v1 (lm-evaluation-harness):

lm_eval --model vllm \
  --model_args pretrained=soyrsoyr/Qwen3.6-27B-FP8-dynamic,dtype=auto,gpu_memory_utilization=0.8,max_model_len=16384,max_num_seqs=256 \
  --tasks arc_challenge --num_fewshot 25 --apply_chat_template --fewshot_as_multiturn --batch_size auto
# ...repeated for hellaswag (10), truthfulqa_mc2 (0), winogrande (5)

Reasoning suite (lighteval), scored pass@1 with the model's own chat template:

lighteval vllm \
  "model_name=soyrsoyr/Qwen3.6-27B-FP8-dynamic,dtype=auto,max_model_length=40960,generation_parameters={temperature:0.6,top_p:0.95,top_k:20,max_new_tokens:32768}" \
  "aime24|0,aime25|0,math_500|0" --save-details