Qwen3.6-35B-A3B-Quark-INT4-G32-GGUF

Qwen3.6-35B-A3B · int4_wo_32 → GGUF Q4_0

Quantization

Reproduce quantization

cd /home/l/work/quantization_work/quark_llama_cpp_test/models/qwen3.6-35b-a3b-q4_0
./scripts/quantize_q4_0.py

This script always uses int4_wo_32. The intermediate Quark checkpoint is written to int4-wo32/ and the final GGUF to gguf-q4_0/. You can override the default paths and device via BASE_MODEL, QUANT_OUT, GGUF_OUT, QUARK_SRC, LLAMA_CPP_DIR, DEVICE, MULTI_GPU=1. To produce only the Quark checkpoint, add --skip-gguf-export.

GGUF export

The target GGUF format for this directory is q4_0, usually exported via the root-level script:

cd /home/l/work/quantization_work/quark_llama_cpp_test
PROFILE=qwen3.6-35b-q4_0 ./run_export.sh

Output is written to gguf-q4_0/ by default.

Evaluation: Perplexity (wikitext-2, llama.cpp)

All three were run with llama.cpp llama-perplexity on wikitext-2 test (n_ctx=512, 20 chunks, same data and same parameters), so they are directly comparable.

Quantization is nearly lossless: Q4_1 is only +1.8% over F16 and Q4_0 only +2.7%, while the size is compressed to about 30% of the float model. Before fixing the linear_attn V-head reordering, Q4_0 PPL was 2,619,116; after the fix it is 6.79.

Evaluation: Open LLM Leaderboard v2

How all 6 tasks are measured on the GGUF

Update (2026-08-06): An earlier version of this card stated that only the 2 generative tasks (IFEval, MATH-hard) could be scored on the GGUF. That limitation has since been lifted by patching llama-server; the four multiple-choice tasks are now measured directly on this GGUF file (see below).

The six leaderboard-v2 tasks use two scoring methods in lm-eval:

• Generative (generate_until): IFEval, MATH-hard. The model generates text freely, which is then matched against rules or the reference answer. Any backend that can generate can score these.
• Multiple-choice (multiple_choice / loglikelihood): BBH, GPQA, MMLU-Pro, MuSR. Each candidate answer is appended to the question, and the model computes the sum of the per-position log-probabilities of that already-fixed token sequence, then picks the highest-scoring candidate. This requires the backend to return the logprob of the input (prompt) tokens.

Stock llama-server returns logprobs only for tokens the model itself generates, not for prompt tokens (both /v1/completions with echo=true and the native /completion with n_probs were tested), so the four multiple-choice tasks originally could not be scored. This was fixed by patching llama-server on branch hongweimeng/gguf-prompt-logprobs: /v1/completions with echo=true + logprobs now returns the legacy OpenAI logprobs format (text_offset / tokens / token_logprobs / top_logprobs) covering the prompt tokens. With that, lm-eval's gguf (GGUFLM) backend scores all four multiple-choice tasks directly on the GGUF. Logprob correctness was verified against llama-perplexity (matching PPL).

So all 6 leaderboard-v2 tasks are now measured on this GGUF file — no vLLM / safetensors substitution is needed.

Results (updated 2026-08-08, all 6 tasks complete)

Backend for this GGUF: the 2 generative tasks (IFEval, MATH-hard) use lm_eval --model local-completions -> llama-server (/v1/completions, -c 32768 -np 4, num_concurrent=4), measured 2026-08-04. The 4 multiple-choice tasks use lm-eval's gguf (GGUFLM) backend against the patched llama-server (branch hongweimeng/gguf-prompt-logprobs, -np 4 -c 131072), measured 2026-08-06. The reference columns were measured with vLLM (--tasks leaderboard, 2026-07-22).

On both the generative and the multiple-choice tasks this GGUF tracks the BF16 baseline closely, indicating faithful quantization export.

¹ Reference only, not measured on this GGUF file (BF16 and vLLM numbers from --tasks leaderboard, 2026-07-22). INT4-GS128 is a group_size=128 sibling model, different from this group_size=32 GGUF; shown for context. ² Multiple-choice / loglikelihood task, measured on this GGUF via the patched llama-server (branch hongweimeng/gguf-prompt-logprobs) + lm-eval gguf backend. BBH/GPQA/MuSR measured 2026-08-06; MMLU-Pro completed 2026-08-08 (~114k loglikelihood requests, 12,032 samples, memory-bandwidth bound at ~0.6 it/s).