Qwen3.6-35B-A3B-int2-mixed-CT-AutoRound

Qwen3.6-35B-A3B 2-bit MoE Mixed AutoRound

Model Details

This model is a INT2/4 bits mixed weight-only quantized version of Qwen/Qwen3.6-35B-A3B, generated with AutoRound.

The checkpoint uses the compressed-tensors WNA16 format for vLLM-compatible inference:

• Attention, linear-attention, and shared-expert projection layers: INT4, group size 128
• Routed MoE expert projection layers: INT2, group size 64
• Activations: BF16 at inference time
• Quantization format: compressed-tensors, pack-quantized

This checkpoint requires compressed-tensors>=0.17.1. Please also follow the license and usage policy of the original base model.

vLLM Inference

This model requires vLLM support for compressed-tensors WNA16 MoE checkpoints with the Humming backend. The support is tracked in vLLM PR #48918.

Until the PR is available in your installed vLLM release, install vLLM from the PR branch:

pip install -v "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/48918/head"
pip install "compressed-tensors>=0.17.1"

Serve the model:

vllm serve <ORG_OR_USER>/<MODEL_ID> \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --served-model-name qwen-mixed-2bit \
  --gdn-prefill-backend triton

For this compressed-tensors WNA16 checkpoint, a supported vLLM build selects the Humming MoE path automatically. Do not pass --moe-backend humming unless your vLLM build explicitly requires it.

Example request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-mixed-2bit",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Summarize mixed-precision MoE quantization in one sentence."}
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

Accuracy

The measured accuracy varies significantly across different backends and evaluation tools and is provided for reference only.

Default settings:

• lmeval hf: add_bos_token=true, max_gen_toks=2048, max_model_len=8192
• lmeval vllm: add_bos_token=true, max_gen_toks=2048, max_model_len=8192
• evalscope vllm: add_bos_token=true, max_tokens=2048, max_model_len=8192

Generate the Quantized Model

The following command follows the AutoRound setup used for this checkpoint:

auto_round \
  --model Qwen/Qwen3.6-35B-A3B \
  --bits 4 \
  --group_size 128 \
  --layer_config "{.*experts.*:{bits:2,group_size:64}}" \
  --enable_alg_ext \
  --enable_torch_compile \
  --iters 1000 \
  --nsamples 512 \
  --format auto_round,llm_compressor \
  --output_dir ./Qwen3.6-35B-A3B-mix

This sets the default quantization scheme to W4G128 and overrides routed expert layers to W2G64.

The command exports both AutoRound format and llm-compressor format. AutoRound format is useful for AutoRound-native workflows and currently supports evaluation through the Hugging Face backend, for example lm_eval --model hf. For faster and more efficient serving/evaluation, use the llm-compressor export, which produces the compressed-tensors checkpoint consumed by vLLM.

Evaluation with EvalScope

EvalScope is the recommended evaluation path for this checkpoint because it evaluates the served vLLM model through the OpenAI-compatible API.

Install EvalScope in the same environment as the compatible vLLM build:

pip install evalscope

Start vLLM:

vllm serve <ORG_OR_USER>/<MODEL_ID> \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --served-model-name qwen-mixed-2bit \
  --gdn-prefill-backend triton \
  --port 8000

Run EvalScope:

evalscope eval \
  --model qwen-mixed-2bit \
  --api-url http://127.0.0.1:8000/v1 \
  --eval-type openai_api \
  --api-key EMPTY \
  --datasets mmlu gsm8k mmlu_pro \
  --generation-config '{"temperature": 0, "max_tokens": 2048, "timeout": 3000}' \
  --eval-batch-size 32 \
  --work-dir ./outputs/evalscope_qwen_mixed_2bit

Optional lm-eval Command

lm_eval --model vllm can be useful for quick comparisons, but for this model card EvalScope is preferred for accuracy reporting.

python -m lm_eval \
  --model vllm \
  --model_args "pretrained=<ORG_OR_USER>/<MODEL_ID>,tensor_parallel_size=1,dtype=bfloat16,max_model_len=8192,gpu_memory_utilization=0.85,trust_remote_code=True,add_bos_token=True,gdn_prefill_backend=triton" \
  --tasks mmlu,gsm8k,mmlu_pro \
  --gen_kwargs "max_gen_toks=2048" \
  --batch_size auto \
  --seed 42 \
  --output_path ./outputs/lmeval_vllm_mixed_2bit

Evaluation Results

We evaluate this quantized checkpoint with EvalScope through a vLLM OpenAI-compatible API. The model is served with the Humming MoE path for compressed- tensors W2A16 checkpoints.

Reproduce the Evaluation

Install the required packages:

pip install -v "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/48918/head"
pip install "compressed-tensors>=0.17.1" evalscope lm-eval

Start vLLM:

```bash
vllm serve Intel/Qwen3.6-35B-A3B-int2-mixed-AutoRound-LLMC \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --served-model-name qwen-mixed-2bit \
  --gdn-prefill-backend triton \
  --port 8000

Run EvalScope:

evalscope eval \
  --model qwen-mixed-2bit \
  --api-url http://127.0.0.1:8000/v1 \
  --eval-type openai_api \
  --api-key EMPTY \
  --datasets mmlu gsm8k mmlu_pro \
  --generation-config '{"max_tokens": 2048, "timeout": 14400}' \
  --eval-batch-size 32 \
  --seed 42 \
  --work-dir ./outputs/evalscope_qwen_mixed_2bit

Optional lm-eval command for quick comparison:

python -m lm_eval \
  --model vllm \
  --model_args "pretrained=Intel/Qwen3.6-35B-A3B-int2-mixed-AutoRound-
  LLMC,tensor_parallel_size=1,dtype=bfloat16,max_model_len=8192,gpu_memory_utilization=0.85,trust_remote_code=True,add_bos_token=True,gdn_prefill_backend=triton" \
  --tasks mmlu,gsm8k,mmlu_pro \
  --gen_kwargs "max_gen_toks=2048" \
  --batch_size auto \
  --seed 42 \
  --output_path ./outputs/lmeval_qwen_mixed_2bit

Limitations

Quantization can change model behavior and may reduce accuracy on some tasks. This checkpoint is optimized for experiments with mixed INT2/INT4 MoE inference and requires a vLLM build with compressed-tensors WNA16 MoE Humming support. Users should run task-specific quality and safety evaluations before deployment.

Ethical Considerations

The model may produce factually incorrect, biased, unsafe, or otherwise undesirable outputs. Downstream users are responsible for validating the model for their use case and for following the base model license and applicable policies.

Citation

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}