Qwen3.6-35B-A3B-Quark-INT4-G128

Qwen3.6-35B-A3B RTN INT4 W4A16 (experts + attn)

Quark INT4 symmetric weight-only (W4A16) quantization of Qwen/Qwen3.6-35B-A3B.

Quantization

Evaluation (Wikitext-2 test)

See eval/ppl_wikitext.json for full results.

Evaluation (Open LLM Leaderboard v2, lm_eval)

Evaluated with EleutherAI/lm-evaluation-harness --tasks leaderboard via vLLM on 2026-07-22. All six leaderboard-v2 tasks were measured on this model. Run config: tensor_parallel_size=4, gpu_memory_utilization=0.90, max_model_len=8192, VLLM_USE_TRITON_AWQ=1.

Per-task raw metrics are shown (the six tasks use different metrics, so no single average is reported). Full result JSON and per-sample JSONL files are in eval/leaderboard/.

Evaluation (GSM8K, 5-shot)

Evaluated alongside the BF16 base model and the UINT4 variant under identical conditions. The prompt suffix <think>\n\n</think>\n\n is prepended to each answer field to suppress the model's thinking preamble so all three runs are comparable (see custom task yaml below).

Eval command:

# 1. Create a custom task yaml that suppresses thinking (save as gsm8k_nothink/gsm8k_nothink.yaml)
cat << 'EOF' > gsm8k_nothink/gsm8k_nothink.yaml
task: gsm8k_nothink
dataset_path: openai/gsm8k
dataset_name: main
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test
doc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
generation_kwargs:
  until:
    - "Question:"
    - "</s>"
    - "<|im_end|>"
  do_sample: false
  temperature: 0.0
repeats: 1
num_fewshot: 5
filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: -1
        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"
      - function: "take_first"
metadata:
  version: 3.0
EOF

# 2. Run evaluation
CUDA_VISIBLE_DEVICES=0 \
VLLM_USE_TRITON_AWQ=1 \
lm_eval run \
  --model vllm \
  --model_args "pretrained=hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.95,enforce_eager=True" \
  --tasks gsm8k_nothink \
  --include_path . \
  --num_fewshot 5 \
  --batch_size auto

VLLM_USE_TRITON_AWQ=1 is required — the default CUDA AWQ kernel enforces OC % group_size == 0, which some MoE expert layers violate at small batch sizes. The Triton kernel has no such restriction.

Comparison with UINT4

vLLM inference

Requires vLLM with Quark W4A16 INT4 support. Set VLLM_USE_TRITON_AWQ=1 to use the Triton AWQ kernel, which handles all layer shapes correctly (see GSM8K note above).

import os
os.environ["VLLM_USE_TRITON_AWQ"] = "1"

from vllm import LLM, SamplingParams

llm = LLM(
    model="hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16",
    quantization="quark",
    dtype="auto",
    gpu_memory_utilization=0.95,
    enforce_eager=True,
)
print(llm.generate(["Hello"], SamplingParams(max_tokens=32))[0].outputs[0].text)

Reproduce

This repo includes scripts/ used to quantize and evaluate:

source scripts/env.sh
bash scripts/01_quantize_rtn_int4.sh   # needs base model + quark + GPU
bash scripts/02_eval_ppl_int4.sh

Base model: Qwen/Qwen3.6-35B-A3B (~67 GB FP16).

Files

• model.safetensors — quantized weights (~21 GB)
• config.json — includes quantization_config with layer excludes
• scripts/ — quantization and PPL evaluation scripts
• eval/ppl_wikitext.json — Wikitext-2 PPL results
• eval/leaderboard/ — Open LLM Leaderboard results (JSON + per-sample JSONL)