Qwen3.6-27B-INT4-W4A16

Qwen3.6-27B-INT4-W4A16

INT4 weight-only (W4A16) quantization of Qwen/Qwen3.6-27B — a dense hybrid model (GatedDeltaNet linear-attention + periodic full-attention) with a vision tower and an MTP head.

Quantization

• AWQ activation-aware scale search + symmetric INT4 weights, group_size 32, MSE observer (llm-compressor).
• Quantized: all MLP, the full-attention self_attn, and the GatedDeltaNet out_proj.
• Kept bf16 (quality-sensitive): GatedDeltaNet in_proj_a/in_proj_b, lm_head, embeddings, vision tower, MTP head.
• Format: compressed-tensors (pack-quantized). Full recipe: recipe.yaml.

Evaluation

Served as W4A16 (vLLM, thinking enabled, temperature 0.6):

Usage (vLLM)

# W4A16 — int4 weights, fp16 activations
vllm serve Avesed/Qwen3.6-27B-INT4-W4A16 \
  --tensor-parallel-size 2 --trust-remote-code --reasoning-parser qwen3

On Ampere the same checkpoint can be served as W4A8 (int4 weights + int8 dynamic activations — faster prefill / batched serving) with the vllm-ampere-optimized fork:

vllm serve Avesed/Qwen3.6-27B-INT4-W4A16 \
  --tensor-parallel-size 2 --marlin-input-dtype int8 --trust-remote-code --reasoning-parser qwen3

Quantized with vllm-ampere-optimized/quantize.