Qwen3.6-35B-A3B-IQ4_K_R4-GGUF

Qwen3.6-35B-A3B — IQ4_K_R4 GGUF (ik_llama.cpp)

Custom IQ4_K_R4 quantization of Qwen/Qwen3.6-35B-A3B, produced with ik_llama.cpp.

This quant is only loadable by ik_llama.cpp — not compatible with mainline llama.cpp.

---

Model Details

• Architecture: Mixture-of-Experts (MoE), 35B total / ~3B active per forward pass
• Quant format: IQ4_K_R4 (ik_llama.cpp-specific)
• File size: ~19 GB
• Source: Re-quantized from Q8_0

---

Hardware Requirements

Fits on a single 16 GB consumer GPU (e.g. RTX 5060 Ti / 5070 Ti) with some CPU offload for MoE layers.

Recommended launch flags:

/path/to/ik_llama.cpp/build/bin/llama-server \
  --model Qwen3.6-35B-A3B-IQ4_K_R4.gguf \
  -ngl 99 --n-cpu-moe 3 -fa 1 -ctk q4_0 -ctv q4_0 \
  -c 131072 --temp 0.6 --jinja --port 8080

--n-cpu-moe 3 offloads 3 expert layers to CPU, keeping VRAM usage within 16 GB while maintaining full GPU utilization for attention and dense layers.

---

Context Length

Native context is 262k tokens. Tested flat throughput to 131k context on 16 GB VRAM.

---

Thinking Mode

Thinking mode is controlled via the enable_thinking parameter:

Via API:

"chat_template_kwargs": {"enable_thinking": false}

Via system prompt (jinja template required): Use the custom template qwen3_nonthinking.jinja to suppress thinking by default.

---

Related

• IQ3_K_R4 version — smaller, ~15 GB, slightly lower quality
• ik_llama.cpp — required inference engine