Downloads · 30 days
0
h1st0ry3D/qwen38-27b-ud-iq2-xxs-mtp
qwen38-27b-ud-iq2-xxs-mtp is a machine learning model from h1st0ry3D. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Benchmarks and llama-swap config for running Qwen3.8-27B at UD-IQ2XXS (9.0 GB) with llama.cpp Multi-Token Prediction (MTP) speculative decoding on a single NVIDIA RTX 4070 12 GB.
Downloads · 30 days
0
Access
Public
Updated Aug 14, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.md2.1 KB · 46%
From the Hugging Face model README
Benchmarks and llama-swap config for running Qwen3.8-27B at UD-IQ2_XXS (9.0 GB) with llama.cpp Multi-Token Prediction (MTP) speculative decoding on a single NVIDIA RTX 4070 12 GB.
The unsloth/Qwen3.8-27B-GGUF → Qwen3.8-27B-UD-IQ2_XXS.gguf file already
contains the embedded NextN/MTP draft head (blk.64.nextn.eh_proj/enorm/hnorm/ shared_head_norm.weight). No separate draft file and no ~50 GB HF-source build
step is required — just serve it with --spec-type draft-mtp.
Verified on llama.cpp b10360:
common_speculative_init_result: creating MTP draft context against the target model
slot print_timing: draft acceptance = 0.71.. (mean len ~3.1)
Drop the block in qwen38-mtp-llama-swap.yaml into your llama-swap.yaml
(models section). Flags: --spec-default --spec-type draft-mtp --spec-draft-n-max 3 --reasoning-preserve.
| ctx | decode tok/s (MTP) | vs no-MTP | peak VRAM |
|---|---|---|---|
| 8K | 68.6 | ~1.8× | 9.86 GB |
| 16K | 68.5 | ~2.0× | 10.02 GB |
| 32K | 69.7 | ~2.2× | 10.48 GB |
Draft acceptance ~0.71 (mean len 3.1). llama-bench (lmx) cannot measure
speculative MTP decode — values are manual llama-server print_timing.
HumanEval (full 164, temp 0.2):
--reasoning auto setting (Qwen3.8 loops at low temp with reasoning on),
not MTP. Use reasoning off for coding.MTP roughly doubles decode but the draft context adds KV. On 12 GB the MTP entry is capped at 32K (10.48 GB) for stability; the non-MTP long-ctx variant reaches 150K.