Qwen3.5-9B-GGUF

Qwen3.5-9B - GGUF (AK line)

👋 I built this line solo - the calibration, the per-tensor allocations, and the eval harness behind every number below. I'm looking for internships in AI agent orchestration and model inference. If this work looks relevant to your team: linkedin.com/in/theaaryankapoor

State-of-the-art GGUF quantizations for Qwen/Qwen3.5-9B. Eight builds (8.95 B params, 32 layers - 24 Gated-DeltaNet + 8 full-attention, GQA 4:1), each with a custom per-tensor bit allocation derived for its size point - plus the stock BF16 vision projector.

Benchmarked head-to-head against every published GGUF of this model within ±3 % of any file below - the Unsloth, bartowski, lmstudio-community, mradermacher, byteshape and AtomicChat lines - every file scored on the same rig against the same BF16 reference: 31 wins, 3 statistical ties, 0 losses across 34 paired comparisons on two held-out evaluation sets. The three ties are the near-lossless Q6/Q8 tiers, where the intervals span zero. Recipe-identical clones (the stock Q8_0 uploads across four publishers - header-verified tensor-for-tensor identical recipes - and static k-quants whose same-size imatrix twin is scored) are cited by class rather than re-scored.

One line per publisher. Log y; the AK line sits below every published line at every size point.

Every comparison with its 95 % interval - blue clears zero, grey is a statistical tie. The full numbers are in the table below.

Comparison set: the widest-distribution GGUF lines for this model as published 2026-08-14; the Method section has what is needed to reproduce any number here.

File naming. Every quant in this line carries the AK- prefix: these are custom per-tensor allocations, not llama.cpp's stock recipes, so AK-Q4_K_M and a stock Q4_K_M are different files. mmproj keeps its upstream name.

Tensor-set class. These are quants of the standard 32-block model - the same class as Unsloth's main repo. bartowski's repo folds in the optional MTP speculative head (~259 MB at Q8_0); in every bartowski comparison below their size is text-tower bytes with the MTP block subtracted, so neither side is charged for weights the other doesn't carry.

Which file do I want?

AK-Q4_K_XL is the strongest file in the crowded 6 GB class - it beats the most-downloaded quant of this model (Unsloth's UD-Q4_K_XL) by 23 % on mean KLD while being smaller, and the margin holds on all six evaluation domains and at 32k context. At the small end the gap widens: AK-IQ2_M beats UD-IQ2_M by 36 % on KLD at identical bytes, and by ~10 points of HumanEval+ pass@1.

llama-server -m Qwen3.5-9B-AK-Q4_K_XL.gguf \
             --mmproj mmproj-BF16.gguf -c 8192 -ngl 99 --jinja

Usage notes (the three things everyone hits): reasoning is off by default on Qwen3.5 small models - enable with --reasoning on (older builds: --chat-template-kwargs '{"enable_thinking":true}'). Set sampling explicitly (upstream ships no generation_config): thinking/general temp 1.0, top_p 0.95, top_k 20, presence_penalty 1.5; precise coding temp 0.6, top_p 0.95, top_k 20. Use --jinja - never --chat-template qwen2. Needs a llama.cpp with qwen35 support (LM Studio ≥ 0.4.6, runtime ≥ v2.5.1); ollama does not run this architecture yet. Vision needs the mmproj file loaded alongside.

Full measurement table

Intervals are a paired per-token cluster bootstrap (100,000 resamples) over the evaluation chunks, computed on held-out text that played no part in building or selecting any file. Δ is against the closest-sized non-AaryanK file (±3 %, same tensor-set class).

Reading the numbers

The Q6/Q8 ties are honest ties. At near-lossless tiers the absolute divergence is tiny (0.003/0.0007) and the intervals span zero; AK-Q8_X trends −16 % against stock Q8_0 at slightly smaller size, but we call it what the interval says.

Raw benchmark scores and fidelity are different claims. Quantization noise flips borderline benchmark answers in both directions; below 4 bpw, multiple-choice accuracy partly measures how quantization perturbs the model's answer-letter prior rather than knowledge. The fidelity metric that tracks actual closeness to BF16 - harmful flip rate on 2,280 MMLU questions - favours or ties the AK file in every size-matched pair (e.g. 4.0 % vs 7.7 % at the IQ3 point, 5.6 % vs 7.2 % at IQ2).

Does the margin generalise?

The flagship re-measured on six evaluation sets: the lead vs UD-Q4_K_XL is interval-clear on all six, including code, Chinese, and chat-templated text. At 32k context the lead holds or grows: −26 % (flagship), −28 % (IQ3 pair), −38 % (IQ2 pair), ahead in every 4k-token position bin.

Capability - measured, not implied

Same serving for every arm (greedy, thinking off), paired statistics throughout:

• MMLU (2,280 q): flagship ties UD-Q4_K_XL (p = 0.45) with fewer harmful flips vs BF16 (1.75 % vs 2.02 %); the IQ3/IQ2 files beat their UD size-peers at p < 1e-4.
• HumanEval+ / MBPP+: flagship pass@1 87.2 / 84.7 - identical to UD and within noise of BF16 (86.0 / 85.7). At IQ2: 84.8 vs UD's 75.0.
• BFCL tool-calling (500-item frozen slice incl. multi-turn agentic): flagship 69.2 % vs UD 69.6 % vs BF16 71.0 % - statistical ties across the board; multi-turn alone: AK matches BF16 (p = 0.85). Zero malformed tool calls in multi-turn on any arm.
• Long-generation stability: 15 fixed prompts × 2,048-token greedy budget on every tier: zero degenerate loops anywhere; cap-hit rates equal to BF16's own.

Tail behaviour

Method

• Reference: our own BF16 GGUF, byte-identical to Unsloth's non-MTP BF16 conversion - the comparison measures quantization, not a conversion delta. llama.cpp pinned at 62bf73d2 for every build and every measurement.
• Eval: llama-perplexity --kl-divergence, ctx 2048, 81,840 scored tokens per file per slice, identical text and reference for every file.
• Statistics: paired per-token cluster bootstrap, 100,000 resamples, per-comparison 95 % intervals, dual-seed stability check; knife-edge results reported as ties.
• Selection/verdict separation: allocations were selected on one instrument and verdicts computed on byte-disjoint held-out text that never touched selection.
• Reproduced rivals: every competitor file was re-scored on this rig (not transcribed from their cards); the UD flagship reproduction was gated against its historical profile before any verdict ran.
• Ship gates: every published artifact rebuilt from scratch with byte-identical tensor payloads; realized per-tensor types audited against intent on every build; inference smoke tests on every tier; capability gates above.
• Scope: the language tower is what is measured and quantized; mmproj ships as the stock BF16 projector. KLD values are model-local - compare within this table only.

---

Per-tensor bit allocation derived separately at each target bit-width using importance data from a diverse in-house calibration set. Base model licence and usage policy unchanged from Qwen/Qwen3.5-9B (Apache-2.0).