Qwen3.8-27B-GPTQ-8bit

Qwen3.8-27B GPTQ 8-bit

GPTQ 8-bit quantization of Qwen/Qwen3.8-27B, a 27B-parameter dense multimodal model. At 8 bits this is effectively lossless — measured perplexity is within noise of the BF16 original — at 60% of the disk/VRAM footprint.

Includes the full vision encoder and MTP (Multi-Token Prediction) module for image understanding and speculative decoding support. A 4-bit FOEM variant (+0.81% perplexity, 21 GB) is available at btbtyler09/Qwen3.8-27B-GPTQ-4bit.

Model Overview

• Architecture: Qwen3_5ForConditionalGeneration (multimodal: text + vision; dense sibling of qwen3_5_moe)
• Total parameters: ~27B
• Layers: 64 (48 linear-attention + 16 full-attention, repeating 3:1 pattern)
• Hidden size: 5120, intermediate size: 17408 (dense MLP — no MoE)
• Context length: 262,144 tokens
• Vision encoder: 27-block ViT, BF16 (333 tensors)
• MTP module: 1-layer speculative decoding head, BF16 (15 tensors)

Quantization Details

All quantizable Linear modules in the text decoder are quantized to INT8 using GPTQ. The vision encoder, MTP module, norms, embeddings, and LM head remain at BF16/FP16.

GPTQ configuration:

• Bits: 8
• Group size: 32
• Symmetric: Yes
• desc_act: No
• true_sequential: Yes
• act_group_aware: Yes
• Fallback: RTN at 0.5% calibration-coverage threshold

Calibration

• Dataset: Mixed — evol-codealpaca-v1 (code) + C4 (general English text)
• Samples: 256, binned uniformly across context lengths 256–2048 tokens
• Quantizer: GPTQModel v6.0.3
• Note: this is general-purpose calibration. Calibrating on wikitext directly would yield lower wikitext perplexity but worse out-of-distribution performance; we optimized for the latter.

The exact quantization script is included in this repo as quantize.py.

Model Size

The total includes the BF16 vision encoder (~1.2 GB) and BF16 MTP head (~0.85 GB) which are kept at full precision.

Perplexity

Evaluated on wikitext-2-raw-v1 (test set), seq_len=2048, stride=512:

Usage

vLLM (Recommended for Serving)

vllm serve btbtyler09/Qwen3.8-27B-GPTQ-8bit \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 262144 \
  --dtype float16 \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image": 2}'

vLLM bug workaround (may apply): Up through at least vLLM 0.19.x, Qwen3_5TextConfig defines ignore_keys_at_rope_validation as a list instead of a set, causing a TypeError during config parsing. Apply this patch before serving if you hit the error:

python3 -c "
for f in [
    '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5.py',
    '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5_moe.py',
]:
    t = open(f).read()
    t = t.replace(
        'ignore_keys_at_rope_validation\"] = [\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        ]',
        'ignore_keys_at_rope_validation\"] = {\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        }')
    open(f,'w').write(t)
    print('Patched', f)
"

Vision Example (via OpenAI API)

import base64, requests

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "btbtyler09/Qwen3.8-27B-GPTQ-8bit",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        {"type": "text", "text": "Describe what you see in this image."},
    ]}],
    "max_tokens": 1024,
})
print(response.json()["choices"][0]["message"]["content"])

GPTQModel / transformers

Loads natively under GPTQModel v6.0.3+ via the upstream Qwen3_5QModel definition (which uses AutoModelForImageTextToText and the multimodal model.language_model.layers.* weight prefix). No checkpoint patching needed.

from gptqmodel import GPTQModel
model = GPTQModel.load("btbtyler09/Qwen3.8-27B-GPTQ-8bit", trust_remote_code=True)

Note: transformers 5.x does not instantiate the MTP submodule for this architecture, so the mtp.* tensors are ignored (with a warning) when loading via transformers/GPTQModel. They are included in the checkpoint for engines that use them for speculative decoding.

Technical Notes

Qwen3.8-27B is a dense multimodal model — it shares the Qwen3_5ForConditionalGeneration wrapper with the MoE-based Qwen3.5/3.6-35B-A3B but uses a standard dense MLP in every decoder layer instead of an expert mixture. The text decoder alternates 3 linear-attention (GatedDeltaNet) layers with 1 full-attention layer, repeated 16 times for 64 total layers.

The vision encoder (27-block ViT) and MTP speculative decoding module are preserved at full BF16 precision from the original model. Only the text decoder's quantizable Linear modules are converted to INT8.

Credits

• Base Model: Qwen — Qwen3.8-27B
• Quantization: GPTQ via GPTQModel v6.0.3
• Quantized by: btbtyler09

License

This model inherits the Apache 2.0 license from the base model.