Downloads · 30 days
80
3% of all-time downloads
hyperhuzaifa/Qwen3.6-35B-A3B-IK-K-GGUF
Qwen3.6-35B-A3B-IK-K-GGUF is a text generation model from hyperhuzaifa. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Self-quantized IKK GGUFs of Qwen/Qwen3.6-35B-A3B — a 35B Mixture-of-Experts (~3B active), so it decodes at small-model speed. Quantized with an importance matrix (qwen35moe.imatrix, included) using ikllama.cpp's IKK q…
Downloads · 30 days
80
3% of all-time downloads
All-time downloads
2.4K
Public
Repo size
72.6 GB
Likes
2
Public
Click a slice to open those files.
.gguf72.5 GB · 100%
From the Hugging Face model README
Self-quantized IK_K GGUFs of Qwen/Qwen3.6-35B-A3B — a 35B Mixture-of-Experts (~3B active), so it decodes at small-model speed. Quantized with an importance matrix (qwen35moe.imatrix, included) using ik_llama.cpp's IK_K quant types, which give noticeably better quality-per-bit than standard K-quants on this MoE.
⚠️ These require ik_llama.cpp. The
IQ*_Ktypes are an ik_llama extension and will NOT load in mainline llama.cpp, LM Studio, Ollama, or koboldcpp. If you need a portable quant, use a standardQ4_K_M/Q5_K_MGGUF instead.
| File | Quant | Size |
|---|---|---|
qwen35moe-IQ4_K.gguf | IQ4_K | 19.7 GB |
qwen35moe-IQ5_K.gguf | IQ5_K | 24.0 GB |
qwen35moe-IQ6_K.gguf | IQ6_K | 28.8 GB |
qwen35moe.imatrix | imatrix used for quantization | 0.11 GB |
IQ4_K fits a single 24 GB GPU with room for context; IQ5_K / IQ6_K are for more VRAM or dual-GPU.
Build ik_llama.cpp, then:
./build/bin/llama-server \
-m qwen35moe-IQ4_K.gguf \
-ngl 99 -fa 1 -c 32768 \
-ctk q8_0 -ctv q8_0 \
-fmoe -rtr \
--jinja --host 127.0.0.1 --port 8080
Flag notes (ik_llama dialect differs from mainline):
-fa 1 — flash attention (not --flash-attn on).-fmoe — ik_llama's fused-MoE kernel; meaningfully faster MoE decode. Recommended for this model.-rtr — run-time tensor repack for your GPU; can speed up decode on some setups (optional).-mtp — speculative decoding regresses on MoE (measured ~−28% on this model). These quants also ship no MTP head. Leave it off.-ub 512 (default) is fastest here; -ub 2048 is ~8% slower on this MoE.Single-stream decode is very fast for a 35B thanks to the ~3B active params — on the order of 175–200+ t/s at IQ4_K on a single modern 24 GB GPU.
Base model: Qwen/Qwen3.6-35B-A3B, Apache-2.0. Quantized locally with ik_llama.cpp + imatrix. Quant types courtesy of ik_llama.cpp.