Qwen3.6-35B-A3B-W4A16

Qwen3.6-35B-A3B-W4A16

An int4 weight-only (W4A16) quantization of Qwen/Qwen3.6-35B-A3B, made with Intel AutoRound and packaged in compressed-tensors for vLLM.

Two things make this build different from the other 4-bit quants of this model:

1. It is calibrated on tool calling and code, not generic web text. Calibration data is what AutoRound uses to decide how to round each weight, so calibrating on pile-10k optimises the rounding for pile-10k.
2. It ships measured expert-coverage statistics (expert_coverage.json). This is an MoE with top-8-of-256 routing; how much of the expert grid your calibration set actually reaches is a real quality variable, and it is normally neither measured nor reported.

Weights are int4, activations stay BF16, so it runs on any Ampere-or-newer NVIDIA GPU through the Marlin int4 kernel. Built and served on RTX 3090s.

Model overview

• Base model: Qwen/Qwen3.6-35B-A3B — a hybrid GatedDeltaNet (linear-attention) + full-attention multimodal MoE: 40 layers (30 linear-attn + 10 full-attn), 256 experts per layer with top-8 routing, ~36B total / ~3B active, plus a vision tower and an MTP head. 262K context.
• Quantization: int4 W4A16, group size 128, symmetric, on the language-decoder Linear layers including all 30,720 expert matrices.
• Format: compressed-tensors (pack-quantized), auto-detected by vLLM — no --quantization flag.
• Quantizer: Intel AutoRound (arXiv:2309.05516), SignRound block reconstruction.

Why this fits two 24 GB cards well

• ~21 GB of weights, so it serves at TP=2 on a pair of 3090s with room to spare.
• ~3B active parameters per token. Decode is bandwidth-bound on the active slice, not on all 36B, so it generates substantially faster than a dense model of similar quality.
• Long context is cheap here. Only 10 of 40 layers keep a KV cache at all, with 2 KV heads at head_dim 256 → ~20 KiB/token at fp16, ~10 KiB at fp8. The full 262144 context costs about 2.6 GB with --kv-cache-dtype fp8. The remaining 30 layers keep a fixed-size recurrent state regardless of sequence length.

Calibration

1024 samples × 2048 tokens, 100 tuning iterations (reduced from 200 after timing the trial run: ~210s/block at 20 iters extrapolated to ~23h for 200 iterations across all 40 blocks on this hardware, over the ~12h budget; 100 iterations was the RUNBOOK's own documented fallback for this case), seed 42.

The general slice is larger than you might expect for a domain-focused build, and it is doing two jobs. The usual one: calibrating on a 100% narrow distribution costs general ability. The MoE-specific one: generic text spreads router mass across more experts. A purely narrow mix concentrates routing on a subset and leaves the tail of the expert grid effectively untuned.

Tool-calling samples are rendered through the model's own chat template with the tool schemas passed in, and Hermes-style JSON tool calls are rewritten into Qwen's XML style, so the calibration text contains the same tokens the model emits at inference.

Expert coverage

Measured over 128 calibration samples, counting top-8 routing hits per (layer, expert) pair:

Full per-layer histograms are in expert_coverage.json in this repo.

Experts in the starved tail receive little tuning signal. On this model's target workload they also receive little inference traffic, so the practical impact is bounded — but it is a real property of any domain-calibrated MoE quant, and it is reported here rather than left implicit.

The BF16 keep-list

Two independent ways to silently ruin this model, both guarded here:

Everything else is quantized, including all mlp.experts.N.{gate,up,down}_proj, mlp.shared_expert.*, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, and the full-attention q/k/v/o.

The keep-list is not passed as glob patterns — AutoRound's layer_config matching is regex, so a leading * silently matches nothing while it quantizes the layer anyway. This build enumerates the real module tree on a meta device, passes exact names, and asserts counts derived from config.json: routers must equal the layer count (40), in_proj_a/b must equal the linear-attention layer count (30), and no expert matrix may appear in the keep-list. "A rule matched something" is too weak a check when one over-match would leave 30,720 expert matrices in BF16.

Deployment (vLLM)

vllm serve fasolack/Qwen3.6-35B-A3B-W4A16 \
    --served-model-name qwen3.6-35b-a3b-w4a16 \
    --tensor-parallel-size 2 \
    --max-model-len 262144 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

• --tool-call-parser qwen3_xml, not hermes — this model emits the XML style.
• TP=2 wants a real interconnect. With only ~3B active parameters the per-layer compute is small, so the all-reduce is a larger share of each step than it would be for a dense model. On NVLink-bridged cards vLLM's custom all-reduce engages and this is fast; straddling PCIe-only cards without P2P gives up much of the benefit.
• If startup OOMs during CUDA-graph capture, add --enforce-eager.

Evaluation

Measured with EleutherAI lm-evaluation-harness against the served endpoint, using the model's recommended sampling.

Note on stop tokens: this model's own reasoning scaffold naturally writes the word "Question" while restating the problem, which collides with GSM8K's (and other tasks') default stop-list (['Question:', '</s>', '<|im_end|>']) and truncates generation almost immediately. Measured with until=<|im_end|> overriding the task default (evaluate.sh's GEN_KWARGS) -- without this override GSM8K measures 3.6-4.4% instead of the real number below. This is a model/harness interaction, not a quantization artifact: reproduced identically against a hand-built single-turn request bypassing lm-eval entirely.

Note on HumanEval methodology: this model's reasoning habit is strong enough that it breaks out of chat-formatted code-completion prompts too -- even lm-eval's own humaneval_instruct task (which pre-seeds the assistant turn inside an open ```python fence) still produced prose instead of code, scoring 0%. Measured instead via local-completions against vLLM's raw /v1/completions endpoint (no chat template at all, direct code continuation) -- the standard way this benchmark was originally designed to run. Spot-checked several failing samples: all were syntactically complete, genuine logic misses (not truncated or malformed output), confirming this is a clean measurement.

Note on HumanEval sampling -- two rows, on purpose: lm-eval's task config sets do_sample: False, but that flag is dead for the API backend used here -- openai_completions.py pops and discards it; only temperature reaches the actual request. So without an explicit override, HumanEval ran with real temperature=1.0 stochastic sampling, one sample per problem (n=1, no averaging) -- a materially riskier setup for code than the deterministic greedy decoding that single-number published pass@1 figures conventionally use, since neither the code nor its decoding got a second chance. Re-ran with temperature=0 (greedy): it recovered ~7 points. GSM8K/MMLU-Pro are far less sensitive to single-sample variance (a stray token rarely invalidates a whole multi-step answer or a multiple-choice pick), so they weren't re-run at greedy.

BF16 baseline measured on a one-off TP=4 rig (this model's own recommended deployment is TP=2 on the NVLink pair, per the note above -- TP=4 was only used here because the 67 GiB BF16 checkpoint needs all four GPUs to fit). Interconnect topology affects serving speed, not the numbers themselves -- these are the actual model outputs, nothing about the comparison is approximate. Getting a usable throughput out of that rig took real debugging; see STATUS.md if you're ever serving this architecture at TP=4 without NVLink yourself.

Reading the table: GSM8K and MMLU-Pro are both close between the two columns, with BF16 very slightly ahead on each -- consistent with what you'd expect from a well-calibrated quantization (small, roughly noise-level cost). HumanEval is the outlier -- and specifically the temperature=1.0 row, not the greedy one: BF16 greedy actually comes out above the quantized build's greedy score (63.41% vs 59.76%), the more apples-to-apples comparison since neither run's decoding has a second chance either way. The temp=1.0 gap (44.51% vs 52.44%) is well within combined stderr (~5.5 pts) for a 164-problem, single-sample benchmark -- read it as sampling noise, not a quantization win, given the greedy numbers point the other way.

Limitations and intended use

• Calibrated narrowly on purpose. Rounding is optimised for coding and tool calling. Outside that distribution it may be marginally worse than a generically-calibrated W4A16 build of the same model. See the expert-coverage section for the MoE-specific dimension of this.
• Weight-only quantization. Activations stay BF16 → maximal compatibility, no activation-quant speedup. On Blackwell hardware an NVFP4 (W4A4) build will be faster.
• The vision tower is untouched and no image data appears in the calibration set (quant_nontext_module=False), so multimodal behaviour should track the base model — but it is not separately evaluated here.
• Inherits the base model's limitations and biases. See the base model card.

Reproducibility

from auto_round import AutoRound

# layer_config is built by enumerating the real module tree on a meta device and
# marking mlp.gate / shared_expert_gate / in_proj_a / in_proj_b / visual.* /
# mtp.* / lm_head as {"bits": 16}, with count assertions against config.json.
ar = AutoRound(
    "Qwen/Qwen3.6-35B-A3B",
    scheme="W4A16",
    dataset=texts,                  # the 1024-sample coding/tool-calling mix above
    nsamples=1024, seqlen=2048, iters=100,  # reduced from 200, see Calibration section
    batch_size=1, gradient_accumulate_steps=4,
    low_gpu_mem_usage=True,
    quant_nontext_module=False,
    trust_remote_code=True, seed=42,
    layer_config=BF16_KEEP_LIST,
)
ar.quantize_and_save(output_dir="Qwen3.6-35B-A3B-W4A16", format="llm_compressor")

Toolchain: auto-round ≥0.15.0, transformers with qwen3_5_moe support, compressed-tensors, torch + CUDA, Python 3.12. Quantized on RTX 3090s.

Citation

@article{cheng2023optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and others},
  journal={arXiv:2309.05516},
  year={2023}
}