Qwen3.8-27B-heretic-ara-INT4-INT8-GPTQ

Qwen3.8-27B-heretic-ara — mixed INT4 + INT8 GPTQ

TL;DR — 22 GB mixed-precision GPTQ checkpoint (INT4 MLP + INT8 attention projections, BF16 for the rest). Measured −0.76pp on GSM8K and +2.5% perplexity vs the BF16 original — i.e. effectively lossless — while cutting weight memory ~2.4× vs BF16. Loads directly in vLLM ≥ 0.27 with quantization=auto_gptq.

A data-driven mixed-precision GPTQ quantization of the heretic-ara fine-tune:

Quantizer: GPTQModel 7.3.4 (data-driven, 607 calibration prompts from the fine-tune's own corpora). The mixed-precision layout is driven by the dynamic field of quantization_config, which both GPTQModel and vLLM ≥ 0.27 parse natively (-:regex = keep unquantized, +:regex = per-module bit override), so the checkpoint is vLLM-loadable as-is.

Quality vs BF16 (teacher-forced full-vocab KLD)

Downstream benchmarks (vLLM 0.27.1, A100 80GB)

Run against the BF16 original with the model's default sampling (temperature=1.0, top_k=20, top_p=0.95), identical prompts/subsets for both.

GSM8K (5-shot, full 1,319-question test set)

The small gap is driven almost entirely by a higher early-EOS rate (the quantized model occasionally stops right after the Answer: prompt at temperature 1.0); conditional on producing a non-empty answer the two models are statistically indistinguishable. If this matters for your workload, lower the temperature (≤ 0.7) or retry empty completions.

Perplexity — C4 validation (30k-doc subset, 293,618 tokens)

Comfortably inside the <5% threshold typically considered safe for aggressive (4-bit) quantization.

Verified on vLLM 0.27.1

• quantization=auto_gptq; mixed 4/8-bit modules resolved per-module (Marlin 4-bit for MLP, 8-bit kernel for the projections)
• text generation, vision (image Q&A), MTP speculative decoding all working (acceptance stats in the run log)

Why the config has fused-name rules

vLLM fuses some separate checkpoint weights on load (qwen3-next-style): in_proj_qkv+in_proj_z → in_proj_qkvz, in_proj_b+in_proj_a → in_proj_ba, q/k/v_proj → qkv_proj, gate/up_proj → gate_up_proj. vLLM resolves each module's precision from the dynamic field using the fused name, so this config adds rules for the fused names (+:.*in_proj_qkvz$ → 8-bit, -:.*in_proj_ba$ → keep BF16, +:.*qkv_proj$ → 8-bit) on top of the per-module GPTQModel rules. Without them vLLM creates the fused GDN module at the base 4-bit width and fails to load the 8-bit shards (shape assert). The weights themselves are exactly the GPTQModel output; only the JSON config was extended.

Serve

vllm serve <this-dir> \
  --dtype bfloat16 --max-model-len 262144 --gpu-memory-utilization 0.92 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}'