Downloads · 30 days
1K
46% of all-time downloads
lmcoleman/Qwen3.6-35B-A3B-BUDGET-GGUF
Qwen3.6-35B-A3B-BUDGET-GGUF is a text generation model from lmcoleman. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
⚠️ Experimental. Use with caution. Both files in this repo are research artifacts from a size-target quantization + QAT experiment. They have been evaluated on perplexity only (all measurements below). No instruction-…
Downloads · 30 days
1K
46% of all-time downloads
All-time downloads
2.2K
Public
Repo size
58 GB
Likes
1
Public
Click a slice to open those files.
.gguf29 GB · 100%
From the Hugging Face model README
⚠️ Experimental. Use with caution. Both files in this repo are research artifacts from a size-target quantization + QAT experiment. They have been evaluated on perplexity only (all measurements below). No instruction-following benchmarks, no agentic evals, no human testing. They are aggressive low-bit builds and will behave worse than the standard tiers. Verify outputs before relying on them.
Derivative of Qwen/Qwen3.6-35B-A3B, compressed to a hard 13.5 GiB byte budget so that the full model plus 64k context fits in 16 GiB of GPU memory (measured on unified memory; see the fit section for what that does and does not prove). If you have more memory, use the standard tiers instead: MagicQuant hybrids (Q4/Q5/Q6) or the ROCmFPX build.
The smallest standard tier of this model is 20.2 GiB (21.7 GB). It cannot fit in 16 GiB at any context length, so this repo asks a different question: what is the best 35B-A3B you can have if the budget is fixed at 16 GiB total, including 64k of context? (Sizes on this card are GiB, 1024-based, throughout.)
The context arithmetic is what makes it plausible at all. Qwen3.6-35B-A3B is a hybrid-attention model: only 10 of its 40 layers are full attention (2 KV heads x 256 head dim); the other 30 are linear-attention layers whose state does not grow with context. KV cache at 64k is therefore about 1.25 GiB at f16, several times smaller than a dense model of this size. 16 GB minus KV minus runtime buffers leaves roughly 13.5 GiB for weights, a 0.20 size ratio versus BF16, well below the Q4 band.
The weights were fitted to that budget with MagicQuant v2's size-target search: an exact per-tensor knapsack (753 tensor assignments, imatrix-calibrated) under a hard byte ceiling, verified against real perplexity rather than a proxy. The file landed at 13.51 GiB against a 13.5 GiB request.
A second copy of the model then went through quantization-aware training (QAT): frozen-mode LoRA trained against this exact quantization layout, merged, and re-packed at the identical 753-tensor allocation. Both copies are published because they win on different workloads (measurements below).
| File | Size | What it is |
|---|---|---|
| Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf | 13.51 GiB | Budget-search output, no QAT |
| Qwen3.6-35B-A3B-BUDGET-13.5GiB-QAT.gguf | 13.51 GiB | Same per-tensor layout, weights adjusted by quantization-aware LoRA |
Which one should you use? The non-QAT file is the recommended default. The QAT file is published as a research artifact: it measured 2.20% better perplexity on held-out chat-formatted data (the distribution it was trained on), while the non-QAT file measured 0.47% better on raw wikitext. Be aware of how weak that evidence is: both deltas sit within about 1.3 standard errors of the reported measurement uncertainty, so the domain split is suggestive, not statistically established. If your workload is chat/agentic and you are willing to run an experimental variant, try the QAT file and judge on your own tasks.
Measured by GPU-memory (GTT) delta on an AMD Strix Halo APU (unified memory), model fully offloaded, 64k context allocated:
| KV cache | Total footprint | Verdict |
|---|---|---|
| f16 | 16.06 GiB | just over |
q8_0 (-fa on -ctk q8_0 -ctv q8_0) | 15.47 GiB | fits |
The 16 GiB claim requires the quantized KV cache flags. Two scope caveats:
All perplexity with llama-perplexity, -c 512. Lower is better.
Out of domain, wikitext-2 (raw encyclopedic prose):
| Build | PPL (Strix Halo, full corpus) | PPL (RTX 3090, 100 chunks) |
|---|---|---|
| BF16 baseline | 6.7208 | n/a |
| BUDGET-13.5GiB | 7.2809 (+8.33% vs BF16) | 7.1882 ± 0.112 |
| BUDGET-13.5GiB-QAT | 7.3150 (+8.84% vs BF16) | 7.2333 ± 0.111 |
In domain, held-out chat blend (297 conversations rendered through the model's own chat template; a row-disjoint resample from the same synthetic generator and bucket weights as the QAT training data, verified 0.00% exact and normalized row overlap; 100 chunks):
| Build | PPL |
|---|---|
| BUDGET-13.5GiB | 2.7127 ± 0.033 |
| BUDGET-13.5GiB-QAT | 2.6529 ± 0.031 (−2.20%) |
Three honest readings of these tables:
This is not a uniform quantization, and it is not one of MagicQuant's standard tiers. The v2 size-target search assigns a scheme to every tensor individually to minimize predicted quality loss under the byte ceiling. What actually ships, by tensor count: 228 F32, 142 Q8_0, 116 Q6_K, 78 Q2_K, 76 Q3_K, 67 Q5_K, 40 F16, 6 Q4_K (753 total). The bulk of the size lives in the MoE expert tensors at Q2_K/Q3_K, which is where most of the quality cost comes from. Norms stay at F32, the router at Q8_0/Q6_K, and the SSM state operands (ssm_a / ssm_dt / conv) at F32/Q8_0; the SSM output projections, by contrast, are among the lowest-precision non-expert tensors in the file (Q2_K/Q4_K), a knapsack outcome, not an oversight.
Search and packing by MagicQuant. Built from the BF16 release weights (not re-quantized from another GGUF).
The MTP draft tensors (blk.40.nextn.*) are present in both files. MTP serving was not measured on this build; it allocates an additional draft context on top of the figures above, and this build's whole premise is ~0.5 GiB of headroom, so budget for that before enabling it. If MTP speculative serving is your goal and you have the memory, the standard tiers document a measured 1.6-1.9x speedup.
llama.cpp (the KV flags are required for the 16 GiB claim; swap in the QAT filename if that is the variant you chose):
llama-server -m Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf \
-c 65536 -fa on -ctk q8_0 -ctv q8_0 -ngl 999 --jinja
LM Studio: untested. These measurements all used a recent llama.cpp build; the qwen3.5/3.6 MoE hybrid-attention architecture needs a runtime new enough to support it. If it loads for you, setting KV cache quantization to Q8_0 is the equivalent of the flags above (advice, not a measurement).
Generated with MagicQuant. Frozen-mode QAT methodology and the controlled experiment behind the domain-split guidance: docs/experiments/qat-frozen-mode-2026-08.md.