Downloads · 30 days
205
4% of all-time downloads
xero0000/Qwen3.6-35B-A3B-vram13-GGUF
Qwen3.6-35B-A3B-vram13-GGUF is a text generation model from xero0000. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 35B MoE that fits entirely in 18 GB of VRAM and decodes at 122–159 t/s on a pair of mid-range gaming GPUs (RTX 3060 Ti 8 GB + RTX 3080 10 GB), with quality within ~1 % perplexity of Q80.
Downloads · 30 days
205
4% of all-time downloads
All-time downloads
4.8K
Public
Repo size
26.5 GB
Likes
2
Public
Click a slice to open those files.
.gguf26.5 GB · 100%
From the Hugging Face model README
A 35B MoE that fits entirely in 18 GB of VRAM and decodes at 122–159 t/s on a pair of mid-range gaming GPUs (RTX 3060 Ti 8 GB + RTX 3080 10 GB), with quality within ~1 % perplexity of Q8_0.
Most sub-24 GB rigs run 35B-A3B MoE models with CPU-offloaded experts and hit a hard RAM-bandwidth wall (~75 t/s on dual-channel DDR4, no matter the GPU). This quant takes the other exit: shrink the model until every weight lives in VRAM. No offload, no -ot, no RAM wall.
| File | Size | What it is |
|---|---|---|
Qwen3.6-35B-A3B-vram13-q2ex-imat.gguf | 12.98 GB | The base quant. Full 256 K context on 18 GB VRAM. |
Qwen3.6-35B-A3B-vram13-q2ex-imat-MTP.gguf | 13.53 GB | Same weights + a transplanted MTP (multi-token-prediction) head for lossless self-speculative decoding. 128 K context, fastest for code. |
| Model | ctx | Code decode | Prose decode | Prefill @14.5 K |
|---|---|---|---|---|
| vram13 | 256 K | 122 t/s | 121 t/s | 1 045 t/s |
| vram13-MTP | 128 K | 159 t/s | 117 t/s | 968 t/s |
| (reference: same model, mixed-quant + CPU-offloaded experts) | 256 K | 74.6 t/s | 74.7 t/s | 673 t/s |
Decode holds >100 t/s even 14.5 K tokens deep. VRAM use: ~6.5 GB + ~8.4–9.4 GB across the two cards.
| Quant | Size | PPL | vs Q8_0 |
|---|---|---|---|
| Q8_0 | 36.9 GB | 2.4053 | — |
| mixed-q2k + imatrix (offload recipe) | 18.2 GB | 2.4195 | +0.6 % |
| vram13 (this repo) | 12.98 GB | 2.4299 | +1.0 % |
| naive q2-expert quant, no imatrix | 12.65 GB | 2.4562 | +2.1 % |
The imatrix + selective layer protection recovers more than half the quality gap of a naive q2 build at nearly the same size.
Requantized from Q8_0 with an imatrix (llama-quantize --allow-requantize):
q2_K on middle layers 3–36, q3_K on the more sensitive first/last three layers (0–2, 37–39)q5_K (this hybrid arch — attention + gated-delta-net SSM — is sensitive here, and these tensors are small)q6_KThe principle: on a MoE with small active experts, expert weights dominate size but tolerate 2-bit with imatrix guidance; everything that touches every token (attention, SSM, shared expert, output) stays at 5–6 bit where the byte cost is negligible.
Qwen3.6's official GGUFs ship without the MTP head, but Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF (an Apache-2.0 coding fine-tune of the same base) kept its trained head. That head was grafted onto this quant by GGUF surgery — append the blk.40 nextn tensors, bump block_count 40→41, set qwen35moe.nextn_predict_layers=1.
Self-speculative decoding verifies every draft against the target model, so output is lossless — only the acceptance rate (speed) depends on how well the donor head predicts this model. On code it accepts well: +30 % decode over the base quant.
Needs ik_llama.cpp (mainline llama.cpp works for the base file; the -mtp flag for the MTP variant is ik-specific).
# base — full 256K window
llama-server -m Qwen3.6-35B-A3B-vram13-q2ex-imat.gguf \
--jinja -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
--ctx-size 262144 -ngl 99 --tensor-split 44,56 --ubatch-size 256 --no-mmap
# MTP — fastest for code (128K is the MTP ceiling)
llama-server -m Qwen3.6-35B-A3B-vram13-q2ex-imat-MTP.gguf \
--jinja -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
--ctx-size 131072 -ngl 99 --tensor-split 44,56 --ubatch-size 256 --no-mmap \
-mtp --ctx-size-draft 8192
Single 16 GB+ GPU: drop --tensor-split. On 2×8 GB it will not fit — use a CPU-offload quant instead.
Qwen3.6 is a reasoning model: it emits chain-of-thought into reasoning_content and the final answer into content — give it generous max_tokens, or disable thinking with "chat_template_kwargs": {"enable_thinking": false}.
nextn_predict_layers should ignore the extra tensors, but mainline behavior isn't guaranteed.Built with TurboQuant/ik_llama.cpp tooling on a Ryzen 5950X + 3060 Ti + 3080. Base model © Alibaba Cloud, Apache-2.0.