Downloads · 30 days
36
1% of all-time downloads
GianniDPC/Qwen3.6-27B-IQ4_NL-with-MTP-GGUF
Qwen3.6-27B-IQ4_NL-with-MTP-GGUF is a text generation model from GianniDPC. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This GGUF combines a smart-mix IQ4NL quantization of Qwen3.6-27B with the MTP (Multi-Token Prediction) head grafted from the unsloth IQ4XS MTP build, enabling native speculative decoding at a compact file size. Optimi…
Downloads · 30 days
36
1% of all-time downloads
All-time downloads
2.7K
Public
Repo size
15.2 GB
Likes
5
Public
Click a slice to open those files.
.gguf15.2 GB · 100%
From the Hugging Face model README
This GGUF combines a smart-mix IQ4_NL quantization of Qwen3.6-27B with the MTP (Multi-Token Prediction) head grafted from the unsloth IQ4_XS MTP build, enabling native speculative decoding at a compact file size. Optimized for 16 GB VRAM GPUs (RTX 4060 Ti 16GB, RTX 4070 Ti Super, RX 7800 XT, etc.).
| File | Size | Quantization |
|---|---|---|
qwen3.6-27b-IQ4_NL-with-MTP.gguf | ~14.19 GB | Smart mix: IQ4_NL + Q4_K + IQ4_XS + F32 |
This is not a pure quantization. It uses a smart mix that assigns different precision levels based on tensor sensitivity:
| Component | Type | Precision | Rationale |
|---|---|---|---|
| Critical layers (FFN gate/down, SSM) | IQ4_NL | High | Non-linear reconstruction preserves quality on sensitive weights |
| Attention QKV projections | Q4_K | Medium | K-quant with per-group scales, good balance for attention |
| FFN up projections | IQ4_XS | Aggressive | Less sensitive, can tolerate tighter compression |
| MTP attention/FFN | Q4_K | Medium | Draft head needs reasonable precision for acceptance |
| MTP FFN up | IQ4_XS | Aggressive | Same rationale as body |
| MTP prediction head (eh_proj) | Q8_0 | Highest | Token prediction is the most sensitive — kept at full 8-bit |
| All norm tensors | F32 | Full | 1D tensors are tiny, no benefit to quantizing |
unsloth/Qwen3.6-27B-MTP-GGUF (IQ4_XS variant), preserving the original mixed quantization for optimal speculative decoding acceptance.| Type | Body Count | MTP Count | Total |
|---|---|---|---|
| IQ4_NL | 370 | 0 | 370 |
| Q4_K | 64 | 6 | 70 |
| IQ4_XS | 64 | 1 | 65 |
| Q8_0 | 0 | 1 | 1 |
| F32 | 353 | 7 | 360 |
| Total | 851 | 15 | 866 |
The Ununnilium pure GGUF strips MTP tensors to save space, but that means speculative decoding can't use the trained native draft head. This file grafts the MTP head back in, restoring native MTP speculative decoding while keeping a smart quantization mix that prioritizes quality where it matters most.
llama-server -hf GianniDPC/Qwen3.6-27B-IQ4_NL-with-MTP-GGUF:IQ4_NL \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0.75
Search for GianniDPC/Qwen3.6-27B-IQ4_NL-with-MTP-GGUF and load the file. Enable speculative decoding in settings with draft-mtp type.
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="GianniDPC/Qwen3.6-27B-IQ4_NL-with-MTP-GGUF",
filename="qwen3.6-27b-IQ4_NL-with-MTP.gguf",
n_gpu_layers=-1,
n_ctx=131072,
speculative="draft-mtp",
speculative_n_draft=3,
)
llama-server \
--model qwen3.6-27b-IQ4_NL-with-MTP.gguf \
--ctx-size 131072 \
--n-gpu-layers 99 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 128 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--threads 6 \
--threads-batch 12 \
--flash-attn on \
--no-mmap \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0.75 \
--temp 0.8 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0
Tested on AMD Radeon RX 7800 XT (16 GB VRAM) with Vulkan backend:
| Metric | Value |
|---|---|
| Cumulative token acceptance | ~82% |
| Draft acceptance rate | ~92% |
| Generation speed | ~42-65 t/s (varies by workload) |
| Prompt processing | ~100-120 t/s |
Acceptance rates above 60% mean MTP provides a net speedup. The ~82% cumulative token acceptance observed in testing indicates strong speculative decoding performance.
| Parameter | Value |
|---|---|
| Architecture | qwen35 |
| Parameters | 27B |
| Layers | 64 + 1 MTP |
| Hidden size | 5120 |
| FFN size | 17408 |
| Attention heads | 24 Q / 4 KV |
| Context length | 262,144 (native) |
| Vocab size | 248,320 |
draft-mtp speculative decoding support (PR #22673 or newer)--parallel 1 is required when using MTP (parallel slots not yet supported)