Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF

Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF

GGUF quantizations of barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP — a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter, and grafted with a Multi-Token Prediction (MTP) head reverse-engineered from the original unpruned checkpoint.

Converted with llama.cpp's convert_hf_to_gguf.py. The MTP head is preserved as native nextn.* / blk.40.* tensors — no manual tensor mapping was required.

A version without the MTP head is available at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF, for backends/setups that don't benefit from self-speculative decoding (see benchmarks below).

Highlights

• +39% to +67% faster generation via self-speculative decoding — measured with llama-bench/llama-cli, 5 runs per config, not estimated. See Benchmarks.
• MTP costs zero measured quality — identical scores to four decimal places vs. the non-MTP release. The speedup above is free. See Quality benchmarks.
• The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss.

Files

All quants ≤Q4_K_M were calibrated with an importance matrix (imatrix) built from 512 samples of barozp/opus-reasoning-distill-train. K-quants (Q*_K*) favor broad compatibility and fast CPU inference; IQ-quants (IQ*) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.

Using the MTP head (speculative decoding)

The GGUF ships the MTP decoder layer as nextn.* (front projection) and blk.40.* (the layer itself). Modern llama.cpp builds can use it for self-speculative decoding — the MTP head drafts a token, the base model verifies it in the same pass.

This is not automatic on every build. Enable it explicitly:

llama-cli -m Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-Q4_K_M.gguf --spec-type draft-mtp -fa on -ngl 99 -p "..."

Without --spec-type draft-mtp (or your frontend's equivalent toggle), generation uses only the base model — behavior and speed then match the non-MTP release.

Recommended draft settings: max draft tokens 3, min draft tokens 0, draft probability 0 (llama.cpp CLI: --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0). Some frontends ship stricter defaults (e.g. LM Studio defaults to draft probability 0.75), which noticeably suppresses acceptance — lowering it to 0 closes most of the gap to the measured rate below.

Measured MTP-head acceptance rate on held-out validation data: ~0.695 (69.5%).

Benchmarks

Measured with llama-cli (Q4_K_M, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM).

Takeaways:

• With speculative decoding off, the MTP file performs identically to the non-MTP file — the extra head carries no measurable VRAM/compute penalty when idle.
• With speculative decoding on, the MTP head gives a substantial, reproducible speedup (39–67% in this environment) — larger at partial GPU offload than at full offload.
• Speculative decoding's benefit depends on whether generation is compute-bound or memory-bandwidth-bound. On systems where the bottleneck is CPU↔GPU memory bandwidth (e.g. heavy CPU offload on a smaller/weaker GPU), the draft+verify overhead can compete with an already-scarce resource and speculative decoding may show little gain or even a small regression. If you observe a slowdown with spec decoding on, disable it (--spec-type none) or use the non-MTP release — both give identical baseline speed.

Quality benchmarks

Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → the non-MTP merge → this checkpoint.

MTP head has zero measured effect on these scores — this checkpoint and the non-MTP release score identically to four decimal places on every task, exactly as expected since transformers never loads the MTP tensors through the standard loading path (see note above). Use this release for the self-speculative-decoding speedup (benchmarks above), not for a quality difference — quality-wise the two releases are interchangeable.

Separately, the reasoning LoRA (present in both releases) pushes ARC-Challenge above the unpruned 256-expert base model (0.616 vs 0.532) — a genuine reasoning-skill transfer from the Opus chain-of-thought training data. MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.

Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the safetensors model card.

Credits

• Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
• Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces (barozp/opus-reasoning-distill-train).
• MTP head: reverse-engineered from the original unpruned checkpoint and grafted onto the pruned+merged model.