Downloads · 30 days
28
5% of all-time downloads
originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY
Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY is a machine learning model from originalGeek. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.
This repo contains only the Qwen3.6-27B MTP / NextN draft head extracted from Unsloth's Q80 MTP GGUF.
Downloads · 30 days
28
5% of all-time downloads
All-time downloads
603
Public
Repo size
451 MB
Likes
3
Public
Click a slice to open those files.
.gguf451 MB · 100%
From the Hugging Face model README
This repo contains only the Qwen3.6-27B MTP / NextN draft head extracted from Unsloth's Q8_0 MTP GGUF.
It is not a standalone chat model. It is a compact graft source for putting the native MTP head back onto a compatible non-MTP Qwen3.6-27B GGUF.
| File | Purpose |
|---|---|
Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf | 15-tensor GGUF containing only blk.64.* MTP tensors |
Local metadata:
tensor_count: 15
qwen35.nextn_predict_layers: 1
qwen35.block_count: 65
mtp_shard.source_block: 64
sha256: F17F5FD3138F7081197E0AFAC75C509F4367099DBDBB2F350CFCBC8017539697
Tensor names:
blk.64.attn_k.weight
blk.64.attn_k_norm.weight
blk.64.attn_norm.weight
blk.64.attn_output.weight
blk.64.attn_q.weight
blk.64.attn_q_norm.weight
blk.64.attn_v.weight
blk.64.ffn_down.weight
blk.64.ffn_gate.weight
blk.64.ffn_up.weight
blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.post_attention_norm.weight
Extracted from:
The upstream Qwen3.6-27B config declares mtp_num_hidden_layers: 1. The head in this file is the Q8_0 GGUF version of that MTP block, placed at blk.64.* for the 64-layer 27B trunk.
MTP is a separate draft module, not just a flag. NVIDIA's MTP docs describe it as extra modules that predict future tokens beyond the normal next-token head. In Qwen3.6-27B, those weights live after the 64 trunk blocks.
Some fine-tune, merge, quantization, or export flows only keep the main model trunk and skip the extra MTP tensors. When that happens, the model can still run normally, but speculative decoding cannot use the trained native draft head. Depending on runtime, MTP may fail to load, silently draft from missing/random weights, or have near-zero acceptance.
Grafting the original MTP head back in can restore native MTP speculative decoding for compatible Qwen3.6-27B derivatives. It is not guaranteed to be optimal after heavy fine-tuning: the draft head was trained against the original trunk hidden states, so acceptance can fall if the tuned trunk diverged a lot.
Compatibility checklist:
qwen35512017408244blk.64.*Do not use this blindly on a model with changed tokenizer, changed dimensions, changed architecture, or missing matching Qwen3.6-27B metadata.
Use a GGUF-aware graft script. Do not concatenate files.
The graft operation needs to:
qwen35.block_count = 65
qwen35.nextn_predict_layers = 1
Conceptual command shape:
python inject_mtp.py `
--body Qwen3.6-27B-non-mtp.gguf `
--mtp-head Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf `
--out Qwen3.6-27B-with-MTP.gguf
After grafting, inspect the result:
$env:PYTHONPATH = "C:\path\to\llama.cpp\gguf-py"
python -m gguf.scripts.gguf_dump Qwen3.6-27B-with-MTP.gguf --no-tensors
python -m gguf.scripts.gguf_dump Qwen3.6-27B-with-MTP.gguf | Select-String "blk.64"
Expected signs:
qwen35.block_count = 65
qwen35.nextn_predict_layers = 1
blk.64.nextn.eh_proj.weight
blk.64.nextn.shared_head_norm.weight
This head can also be grafted onto a compatible Qwen3.6-40B derivative when that model is a depth-expanded descendant of the same Qwen3.6-27B family, as with the local IQ4_XS graft. Treat this as an experimental compatibility path, not as a native 40B MTP head.
The important difference is layer count. The original Qwen3.6-27B body has 64 main layers, so this donor head is stored as blk.64.*. The expanded 40B body used locally has 96 main layers, so the grafted MTP block belongs after those layers as blk.96.*.
For a 96-layer 40B body, the graft script needs to:
qwen35 and is dimensionally compatible.blk.64.* to blk.96.* while appending them.qwen35.block_count = 97
qwen35.nextn_predict_layers = 1
Conceptual command shape:
python inject_mtp.py `
--body Qwen3.6-40B-IQ4_XS-non-mtp.gguf `
--mtp-head Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf `
--out Qwen3.6-40B-IQ4_XS-with-MTP.gguf `
--target-mtp-block 96
After grafting, inspect for blk.96 rather than blk.64:
$env:PYTHONPATH = "C:\path\to\llama.cpp\gguf-py"
python -m gguf.scripts.gguf_dump Qwen3.6-40B-IQ4_XS-with-MTP.gguf --no-tensors
python -m gguf.scripts.gguf_dump Qwen3.6-40B-IQ4_XS-with-MTP.gguf | Select-String "blk.96"
Expected signs:
qwen35.block_count = 97
qwen35.nextn_predict_layers = 1
blk.96.nextn.eh_proj.weight
blk.96.nextn.shared_head_norm.weight
The reason this works structurally is that the expanded 40B variant keeps the same core tensor shapes as Qwen3.6-27B: hidden size 5120, attention heads 24, KV heads 4, and FFN width 17408. Those matching dimensions make the MTP head tensors loadable after remapping from block 64 to block 96.
That does not mean quality or acceptance is guaranteed. The donor MTP head was trained to predict from Qwen3.6-27B's layer-64 hidden state, while a 96-layer 40B graft feeds it the representation after layer 96. If draft acceptance is poor, lower --spec-draft-n-max to 3 or 1 and benchmark again. A truly optimal 40B MTP head would need to be trained against the 40B trunk.
MTP support for Qwen3.6 GGUF has moved past the original PR #22673 merge notes. A useful build now needs all of these pieces:
--spec-type draft-mtp: https://github.com/ggml-org/llama.cpp/pull/22673--spec-draft-type-k and --spec-draft-type-v are honored: https://github.com/ggml-org/llama.cpp/pull/23646turbo2, turbo3, and turbo4, if you want the TurboQuant profiles documented below.Still-current flag notes:
--spec-type draft-mtp, not the older --spec-type mtp.--spec-draft-type-k and --spec-draft-type-v to mirror the target KV cache type you actually want for the MTP draft path.--spec-draft-n-max 6 is the normal live depth for 27B Qwen3.6 native-MTP profiles here.--spec-draft-p-min 0.0 for throughput. 0.75 remains a useful stricter acceptance / regression fallback from the Unsloth guidance.--spec-type ngram-mod,draft-mtp.Related external notes:
Fresh check on 2026-05-28: ggml-org/llama.cpp master was at b9371-1-g8ad8aef44. No newer MTP, NextN, NVFP4, FWHT, or TurboQuant change was better than the tested vs7 stack below. The most relevant newer CUDA change only gates PDL for old CUDA Toolkit versions; this build uses CUDA 13.1.
The old hand-merge recipe for "PR #22673 plus TurboQuant" has been superseded locally. The current tested Windows / RTX 5090 runtime is:
repo: llama-cpp-turboquant-MTP-NVFP4-13.1-ngram
branch: tq-mtp-refresh-20260526-vs7-fwht
commit: ea82a5bda
llama.cpp version: 9432
build: build-cuda131-sm120-vs7-fwht
binary: build-cuda131-sm120-vs7-fwht\bin\Release\llama-server.exe
That vs7 runtime combines TheTom's TurboQuant KV-cache branch with the current MTP cleanup stack, the MTP draft-KV type fix, NVFP4 MTP scale tensor support, and CUDA FWHT support. In practice, this means the local tree no longer needs the old conflict-resolution recipe in this README; use the checked-out vs7 branch or recreate that patch stack directly.
Feature sanity checks:
.\build-cuda131-sm120-vs7-fwht\bin\Release\llama-server.exe --help |
findstr /C:"draft-mtp" /C:"ngram-simple" /C:"turbo4"
.\build-cuda131-sm120-vs7-fwht\bin\Release\test-backend-ops.exe test -b CUDA0 -o MUL_MAT_HADAMARD
For Blackwell / RTX 5090 we used CUDA 13.1, not CUDA 13.2. The Unsloth discussion and guide warn that CUDA 13.2 can produce garbled output in this area; the community note says to use CUDA 13.1 or lower until NVIDIA fixes it in a later toolkit.
The tested local build used:
nvcc: Cuda compilation tools, release 13.1, V13.1.80
MSVC: 19.44
CMake: Visual Studio 17 2022 generator
CUDA arch: 120, normalized by CMake/CUDA to 120a
The local build tree was:
llama-cpp-turboquant-MTP-NVFP4-13.1-ngram
branch: tq-mtp-refresh-20260526-vs7-fwht
commit: ea82a5bda
describe: feature-turboquant-kv-cache-b9410-2b61ea2-22-gea82a5bda
included fixes: llama.cpp PR #23646, #23563, #23615, #23690
latest upstream checked: ggml/master b9371-1-g8ad8aef44 on 2026-05-28
Build command shape:
$cuda = "C:\Users\Lui\Documents\GitHub\Local_llamacpp\cuda-13.1-toolkit"
$repo = "C:\Users\Lui\Documents\GitHub\Local_llamacpp\llama-cpp-turboquant-MTP-NVFP4-13.1-ngram"
$env:CUDA_PATH = $cuda
$env:CUDA_PATH_V13_1 = $cuda
$env:PATH = "$cuda\bin;$cuda\bin\x64;$env:PATH"
cmake -S $repo -B "$repo\build-cuda131-sm120-vs7-fwht" `
-G "Visual Studio 17 2022" -A x64 `
-T "cuda=$cuda" `
-DGGML_CUDA=ON `
-DGGML_CUDA_FA_ALL_QUANTS=ON `
-DGGML_CUDA_GRAPHS=ON `
-DCMAKE_CUDA_ARCHITECTURES=120 `
-DCUDAToolkit_ROOT="$cuda" `
-DLLAMA_CURL=OFF `
-DLLAMA_BUILD_UI=OFF
cmake --build "$repo\build-cuda131-sm120-vs7-fwht" `
--config Release `
--target llama-server llama-bench test-backend-ops `
--parallel 12
Copy CUDA 13.1 runtime DLLs beside the binaries if needed:
Copy-Item "$cuda\bin\x64\*.dll" "$repo\build-cuda131-sm120-vs7-fwht\bin\Release" -Force
NVFP4 + true TurboQuant KV + MTP, live coding-agent default:
set TURBO_AUTO_ASYMMETRIC=0
llama-server.exe `
--model Qwen3.6-27B-with-MTP-or-native-MTP-NVFP4.gguf `
--ctx-size 262144 `
--n-gpu-layers 999 `
--main-gpu 0 `
--parallel 1 `
--threads 8 `
--threads-batch 8 `
--batch-size 2048 `
--ubatch-size 512 `
--cache-type-k turbo4 `
--cache-type-v turbo4 `
--spec-draft-type-k turbo4 `
--spec-draft-type-v turbo4 `
--flash-attn on `
--poll 100 `
--cache-ram 16384 `
--spec-type draft-mtp `
--spec-draft-n-max 6 `
--spec-draft-p-min 0.0 `
--host 127.0.0.1 `
--port 8080 `
--metrics
Replay/copy-heavy variants:
--spec-type ngram-mod,draft-mtp
# or targeted benchmark mode:
--spec-type ngram-simple,draft-mtp
Stricter MTP acceptance fallback:
--spec-draft-p-min 0.75
On an RTX 5090 with the local CUDA 13.1 build and a native-MTP NVFP4 Qwen3.6-27B GGUF, historical targeted sweeps showed:
Repeated-context copy workload, 1024 generated tokens:
| Spec type | TG | Note |
|---|---|---|
draft-mtp | 188.70 t/s | baseline MTP |
ngram-simple,draft-mtp | 804.12 t/s | best tested replay speed |
ngram-map-k4v,draft-mtp | 781.82 t/s | close second |
ngram-mod,draft-mtp | 689.76 t/s | good, but not best here |
ngram-map-k,draft-mtp | 502.60 t/s | unstable in this sample |
ngram-cache,draft-mtp | 152.96 t/s | not useful for this setup |
Long-context coding-agent style workload, about 86k prompt tokens, 1024 generated tokens:
| Spec type | TG |
|---|---|
draft-mtp | 66.78 t/s |
ngram-simple,draft-mtp | 70.70 t/s |
ngram-map-k4v,draft-mtp | 58.84 t/s |
ngram-mod,draft-mtp | 55.75 t/s |
For this machine/build, the live 27B coding-agent profiles currently stay on plain draft-mtp. Ngram + MTP remains useful for replay/copy-heavy prompts and should be selected per workload, not treated as a universal default. The reduced vs7 correctness smoke on the Unsloth 27B MTP profile initialized draft-MTP with draft KV q8_0/turbo3 and accepted 9 of 12 draft tokens.