Downloads · 30 days
42
16% of all-time downloads
bogdan-radulescu/qwen3.6-35B-A3B-asym-GGUF
qwen3.6-35B-A3B-asym-GGUF is a text generation model from bogdan-radulescu. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
A GGUF quantization of Qwen3.6-35B-A3B built for larger-than-RAM / SSD-streaming inference. Rather than quantizing every tensor to the same width, it quantizes asymmetrically: the routed-expert weights (the bulk of th…
Downloads · 30 days
42
16% of all-time downloads
All-time downloads
266
Public
Repo size
11.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf11.7 GB · 100%
From the Hugging Face model README
A GGUF quantization of Qwen3.6-35B-A3B built for larger-than-RAM /
SSD-streaming inference. Rather than quantizing every tensor to the same width,
it quantizes asymmetrically: the routed-expert weights (the bulk of the bytes
moved per token in a Mixture-of-Experts model) are pushed to ~2 bits, while
everything quality-sensitive (attention, router, shared experts, embeddings,
norms) stays at Q8_0.
It matches the 4-bit IQ4_XS baseline on quality (identical 9/10 coding) while being ~33% smaller and streaming fewer cold bytes per token.
This quant is built for Fono, whose aim is to run the most capable model that will actually fit on ordinary, memory-limited hardware. The guiding principle is intelligence per gigabyte: spend precision only where the model is genuinely sensitive, and compress everything else hard. That lets a larger, smarter model drop into the same memory budget, so a modest machine gets to punch well above its weight instead of settling for a smaller, weaker model.
In an MoE, decode speed under memory pressure is dominated by how many routed-expert bytes must be pulled from SSD per token. So this recipe spends bits where they matter and saves them where they don't:
| Tensor group | Precision | Why |
|---|---|---|
Routed experts ffn_gate_exps / ffn_up_exps | IQ2_XXS (~2.06 bpw, imatrix) | dominant streamed mass — the big lever |
Routed expert ffn_down_exps | Q2_K (imatrix) | smart 2-bit; dims are natively ÷256, so no padding needed |
Attention / router / shared experts (ffn_*_shexp) / embeddings / norms | Q8_0 | near-lossless where the model is sensitive |
Note the architectural contrast with the gemma-4 sibling: Qwen uses separate
gate/up/down expert tensors plus always-on shared experts, and its routed
reduction dims are already 256-aligned (down_exps = 512, gate/up = 2048).
So the smart imatrix i-quants tile directly — no zero-padding required. This
is what makes the technique general: the only arch-specific wrinkle (alignment)
is a no-op here.
Only imatrix-aware i-quants (IQ2_XXS, Q2_K) are used for the 2-bit tensors;
crude legacy 2-bit formats at the same bit budget collapse model capability. The
importance matrix was computed with llama-imatrix over Bartowski's
calibration_datav3 (generic web/wiki text, deliberately disjoint from our
benchmark tasks so scores aren't inflated), 100 chunks × 512 tokens.
There are two distinct wins:
Other reasons: a Qwen3.6 GGUF that keeps full coding capability at ~⅔ the disk of the 4-bit build, and low bytes-moved-per-token rather than just a small file.
(If the full-size model already fits comfortably in your RAM, a uniform 4-bit build is marginally faster there — fewer dequant ops. The asymmetric quant's advantage is exactly the common case where it fits and the bigger one doesn't.)
Coding = run-the-code pass rate (10 objective tasks); polish = instruction-formatting score.
| variant | size | coding | polish | note |
|---|---|---|---|---|
| IQ4_XS (uniform 4-bit baseline) | 17.7 GB | 10/10 | 0.986 | reference |
| asym 2-bit (this model) | 11.74 GB | 9/10 | 0.981 | −33% size, fewer cold bytes/token, faster @6G cap |
The single lost coding point is the same hard Rust task the baseline's asym-class peers also miss; general prose, polish, and the other 9 tasks are unaffected.
qwen3.6-35b-a3b-asym.gguf (11.74 GB)llama-cli -m qwen3.6-35b-a3b-asym.gguf -p "Write a prime sieve in Rust."