Downloads · 30 days
12
0% of all-time downloads
GianniDPC/Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF
Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF is a text generation model from GianniDPC. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This GGUF combines the Ununnilium IQ4XS pure quantization of Qwen3.6-27B with the MTP (Multi-Token Prediction) head extracted from the unsloth IQ4XS MTP build, enabling native speculative decoding at a compact file size.
Downloads · 30 days
12
0% of all-time downloads
All-time downloads
10.3K
Public
Repo size
14.6 GB
Likes
23
Public
Click a slice to open those files.
.gguf14.6 GB · 100%
From the Hugging Face model README
This GGUF combines the Ununnilium IQ4_XS pure quantization of Qwen3.6-27B with the MTP (Multi-Token Prediction) head extracted from the unsloth IQ4_XS MTP build, enabling native speculative decoding at a compact file size.
| File | Size | Quantization |
|---|---|---|
qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf | ~13.57 GB | Body: IQ4_XS, MTP: IQ4_NL/Q5_K/Q8_0 mix |
llama-quantize --pure from Ununnilium's build, using the unsloth imatrix for calibrationunsloth/Qwen3.6-27B-MTP-GGUF (IQ4_XS variant), preserving the original mixed quantization:
attn_q, attn_k, attn_output, ffn_gate/up/down → IQ4_NLattn_v → Q5_Knextn.eh_proj → Q8_0The Ununnilium pure GGUF strips MTP tensors to save space, but that means speculative decoding can't use the trained native draft head. This file grafts the MTP head back in, restoring native MTP speculative decoding while keeping the aggressive IQ4_XS body quantization for VRAM efficiency.
llama-server -hf GianniDPC/Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF:IQ4_XS \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0.75
Search for GianniDPC/Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF and load the file. Enable speculative decoding in settings with draft-mtp type.
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="GianniDPC/Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF",
filename="qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf",
n_gpu_layers=-1,
n_ctx=131072,
speculative="draft-mtp",
speculative_n_draft=3,
)
llama-server \
--model qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf \
--ctx-size 131072 \
--n-gpu-layers 99 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 128 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--threads 6 \
--threads-batch 12 \
--flash-attn on \
--no-mmap \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0.75 \
--temp 0.8 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0
Tested on AMD Radeon RX 7800 XT (16 GB VRAM) with Vulkan backend:
| Metric | Value |
|---|---|
| Cumulative token acceptance | ~82% |
| Draft acceptance rate | ~92% |
| Generation speed | ~42-65 t/s (varies by workload) |
| Prompt processing | ~100-120 t/s |
Acceptance rates above 60% mean MTP provides a net speedup. The ~82% cumulative token acceptance observed in testing indicates strong speculative decoding performance.
| Parameter | Value |
|---|---|
| Architecture | qwen35 |
| Parameters | 27B |
| Layers | 64 + 1 MTP |
| Hidden size | 5120 |
| FFN size | 17408 |
| Attention heads | 24 Q / 4 KV |
| Context length | 262,144 (native) |
| Vocab size | 248,320 |
draft-mtp speculative decoding support (PR #22673 or newer)--parallel 1 is required when using MTP (parallel slots not yet supported)