Qwen3-Coder-30B-A3B-Instruct-W4A16

Qwen3-Coder-30B-A3B-Instruct-W4A16

W4A16 (INT4 group-128 weights + FP16 activations) quantization of Qwen/Qwen3-Coder-30B-A3B-Instruct.

• Quantized with llm-compressor on an NVIDIA H200, in the compressed-tensors pack-quantized format.
• Designed for inference on 2× NVIDIA A2 (16 GB, Ampere SM 8.6) with vLLM (tensor-parallel across the two cards).

The point of this build is to fit this 30B-A3B MoE onto small, FP8-less GPUs like the A2, where BF16 (~57 GB) and INT8 (~30 GB) don't fit. At 4-bit the checkpoint is ~16 GB (~8 GB/GPU at TP=2), running via the Marlin INT4 kernel — which, unlike FP8 / W4AFP8, works on Ampere.

What's quantized

Only transformer Linear weights are quantized; the embedding, output head, router gates, and norms stay BF16 for quality. It remains a standard Qwen3MoeForCausalLM — full GQA attention, SwiGLU, 128 experts / 8 active — so it uses vLLM's mainstream MoE path.

Serving with vLLM

vllm serve systemsofrecord/Qwen3-Coder-30B-A3B-Instruct-W4A16 \
  --tensor-parallel-size 2 \
  --dtype float16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

• No FP8 required — runs on Ampere (A2 / A10 / A30 / …) and newer.
• KV cache is FP16 (Ampere has no FP8 KV); GQA (4 KV heads) keeps it small.
• Needs a vLLM build with Qwen3MoeForCausalLM support (≥ 0.25).

Verified: loaded and generated correct code on 2× NVIDIA A2 under vLLM 0.25.1 — ~7.9 GB weights/GPU at TP=2, Marlin wNa16 MoE kernel, CUDA graphs captured cleanly.

Quantization recipe

• Tool: llm-compressor (run on an NVIDIA H200).
• Scheme: W4A16 — weights 4-bit int, group_size=128, symmetric; activations unquantized.
• Method: model-free RTN (round-to-nearest) weight quantization.
• Format: pack-quantized (INT4 packed into INT32 + group scales).
• Ignore (BF16): lm_head, embed_tokens, MoE router gates, norms.
• Target: 2× NVIDIA A2 served with vLLM (TP=2).

License & attribution

Apache-2.0, inherited from the base model Qwen/Qwen3-Coder-30B-A3B-Instruct. This repository only redistributes a quantized copy of those weights; all model capabilities and credit belong to the Qwen team.