Downloads · 30 days
152
12% of all-time downloads
phucngodev/Qwen3.6-27B-MTP
Qwen3.6-27B-MTP is a text generation model from phucngodev. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;border:1px solid c7d2fe;border-radius:18px;overflow:hidden;background:ffffff;margin-bottom:28px;box-shadow:0 6px 20px -12px rgba(79…
Downloads · 30 days
152
12% of all-time downloads
All-time downloads
1.2K
Public
Repo size
230 GB
Likes
0
Public
Click a slice to open those files.
.gguf229 GB · 100%
From the Hugging Face model README
<div style="display:grid;grid-template-columns:repeat(2,1fr);gap:14px;margin-bottom:32px;"> <div style="display:flex;gap:14px;align-items:flex-start;border:1px solid #e2e8f0;border-radius:14px;padding:18px;background:#ffffff;"> <div style="flex:0 0 44px;height:44px;border-radius:12px;display:flex;align-items:center;justify-content:center;background:linear-gradient(135deg,#eef2ff 0%,#ecfeff 100%);font-size:20px;border:1px solid #c7d2fe;">⚡</div> <div> <div style="font-size:13.5px;font-weight:700;color:#0f172a;margin-bottom:4px;">MTP Decoding</div> <div style="font-size:12.5px;color:#475569;line-height:1.55;">Multi-Token Prediction drafts future tokens and accepts them when the main decode path agrees — cutting wall time on long reasoning, code generation, and tool-call setup.</div> </div> </div> <div style="display:flex;gap:14px;align-items:flex-start;border:1px solid #e2e8f0;border-radius:14px;padding:18px;background:#ffffff;"> <div style="flex:0 0 44px;height:44px;border-radius:12px;display:flex;align-items:center;justify-content:center;background:linear-gradient(135deg,#fff7ed 0%,#fef3c7 100%);font-size:20px;border:1px solid #fed7aa;">🧩</div> <div> <div style="font-size:13.5px;font-weight:700;color:#0f172a;margin-bottom:4px;">No-Thinking by Design</div> <div style="font-size:12.5px;color:#475569;line-height:1.55;">Trained on Qwen3.6's non-thinking inference path. The model responds directly with tool calls, edits, and structured output — no <code><thinking></code> preamble eating wall time before the harness can act.</div> </div> </div> <div style="display:flex;gap:14px;align-items:flex-start;border:1px solid #e2e8f0;border-radius:14px;padding:18px;background:#ffffff;"> <div style="flex:0 0 44px;height:44px;border-radius:12px;display:flex;align-items:center;justify-content:center;background:linear-gradient(135deg,#ecfeff 0%,#f0fdfa 100%);font-size:20px;border:1px solid #a5f3fc;">🧪</div> <div> <div style="font-size:13.5px;font-weight:700;color:#0f172a;margin-bottom:4px;">Local-First Throughput</div> <div style="font-size:12.5px;color:#475569;line-height:1.55;">Designed for llama.cpp-class runtimes on a single workstation, with MTP draft acceptance measured on real agent workloads.</div> </div> </div> <div style="display:flex;gap:14px;align-items:flex-start;border:1px solid #e2e8f0;border-radius:14px;padding:18px;background:#ffffff;"> <div style="flex:0 0 44px;height:44px;border-radius:12px;display:flex;align-items:center;justify-content:center;background:linear-gradient(135deg,#fdf2f8 0%,#f5f3ff 100%);font-size:20px;border:1px solid #fbcfe8;">🚀</div> <div> <div style="font-size:13.5px;font-weight:700;color:#0f172a;margin-bottom:4px;">Practical Speed</div> <div style="font-size:12.5px;color:#475569;line-height:1.55;">Built for workflows where waiting minutes per turn breaks the loop — the tune favors decisive output over scratch-pad expansion.</div> </div> </div> </div> <div style="display:grid;grid-template-columns:repeat(2,1fr);gap:14px;margin:0 0 32px 0;"> <div style="border:1px solid #c7d2fe;border-left:5px solid #4338ca;border-radius:14px;padding:18px 22px;background:linear-gradient(135deg,#ffffff 0%,#eef2ff 100%);"> <div style="font-size:10.5px;font-weight:800;color:#4338ca;text-transform:uppercase;letter-spacing:0.14em;">Context Window</div> <div style="font-size:30px;font-weight:800;color:#0f172a;margin-top:6px;line-height:1;">128k <span style="font-size:13px;font-weight:600;color:#64748b;">tested</span></div> <div style="font-size:12px;color:#475569;margin-top:8px;line-height:1.5;"><b>256k</b> natively supported by the base · extensible up to <b>1M</b> tokens via RoPE scaling.</div> </div> <div style="border:1px solid #fbcfe8;border-left:5px solid #be185d;border-radius:14px;padding:18px 22px;background:linear-gradient(135deg,#ffffff 0%,#fdf2f8 100%);"> <div style="font-size:10.5px;font-weight:800;color:#be185d;text-transform:uppercase;letter-spacing:0.14em;">MTP Draft Acceptance</div> <div style="font-size:30px;font-weight:800;color:#0f172a;margin-top:6px;line-height:1;">~78<span style="font-size:18px;font-weight:700;color:#64748b;">%</span></div> <div style="font-size:12px;color:#475569;margin-top:8px;line-height:1.5;">Drafted future tokens accepted by the main decode path on agent workloads. <b>3 speculative steps · 4 draft tokens</b>.</div> </div> </div>[!TIP] For the strongest Pi-style coding-agent behavior, use the reasoning-trained release:
bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF. See the technical writeup for the broader evaluation context. This no-thinking tune remains useful when you specifically want the lower-latency direct / instruct path.
| Attribute | Details |
|---|---|
| Base model | Qwen/Qwen3.6-27B |
| Release format | GGUF |
| Runtime target | llama.cpp-compatible local inference |
| Tuning focus | Harness fluency, coding-agent tasks, terminal workflows, tool use, repository work |
| Fine-tuning style | 4-bit QLoRA SFT on private passed agent trajectories |
| Technical writeup | Qwen3.6 27B reasoning writeup |
| Reasoning data policy | Internal reasoning traces were not exported into the SFT rows |
| Recommended quant | Q4_K_M as the default starting point |
[!NOTE] Qwen3.6-27B is a causal language model with a vision encoder. Image and video understanding are supported by pairing the language-model GGUF with the compatible Qwen3.6
mmproj-F16.ggufsidecar (see §4 Multi-modal inference). The MTP draft heads are kept atQ8_0precision inside every quant via--tensor-type nextn=q8_0at quantize time — speculative decoding works at any quant level, not justQ8_0/bf16.
Qwen3.6 supports both a thinking inference mode (which emits a <thinking>...</thinking> block before responding) and a non-thinking mode (where the model answers and acts directly). This release is fine-tuned specifically for the no-thinking path — the mode where agent loops actually live. In a PI-style harness running tool-call loops, every thinking token is wall time the harness can't dispatch the next action against, so the tune is shaped to make the no-thinking path quality where it counts: tool calls, repository edits, terminal commands, verifier feedback, and structured output.
This carries forward Qwen3.6's existing agentic-coding posture — frontend workflows, repository-level reasoning, and tool calling — but pulls the quality into the inference mode that local agent runtimes can budget for.
Recommended starting point: Q4_K_M.
| Quant | File size | VRAM (approx) | Suggested use |
|---|---|---|---|
Q2_K | ~11 GB | ~13 GB | Smallest memory footprint; quality tradeoffs are expected. |
Q3_K_S | ~12 GB | ~15 GB | Low-memory 3-bit option. |
Q3_K_M | ~14 GB | ~16 GB | Balanced 3-bit option. |
Q3_K_L | ~15 GB | ~17 GB | Higher-quality 3-bit option. |
Q4_K_S | ~16 GB | ~18 GB | Smaller 4-bit option. |
Q4_K_M | ~17 GB | ~19 GB | Default recommendation for most local use. Comfortable on a 24 GB GPU. |
Q5_K_S | ~19 GB | ~21 GB | Higher-quality 5-bit option. |
Q5_K_M | ~20 GB | ~22 GB | Strong quality/memory tradeoff; near the upper edge of a 24 GB GPU. |
Q6_K | ~22 GB | ~25 GB | High-quality local inference if you have the memory. |
Q8_0 | ~29 GB | ~32 GB | Highest-precision quantized option. |
bf16 | ~55 GB | ~58 GB | BF16 GGUF reference, if present. |
VRAM figures are rough estimates for GPU-offload inference (-ngl 99 -fa) at a moderate context (~32k) with quantized KV cache; they scale up with longer contexts.
[!IMPORTANT] Every quant in this release ships with the MTP
nextnprediction heads stored atQ8_0precision, regardless of the overall quant target. That means speculative decoding works at any quant level — pick the smallest one that fits your VRAM and you still get the MTP throughput profile described in §6.
Some files may still be uploading. Check the Files tab for the exact artifacts currently available.
Run with llama.cpp (standard launch — works on any build):
# 128k context shown; base model natively supports 256k and is extensible to ~1M via RoPE scaling.
# Sampling values match Qwen3.6's recommended non-thinking-mode defaults — this is the inference
# path the tune was trained for, so prefer these.
llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M \
--jinja -ngl 99 -fa -c 131072 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
--presence-penalty 1.5
Run with upstream llama.cpp + MTP speculative decoding (ggml-org/llama.cpp, MTP support merged in PR #22673):
# The nextn prediction heads in this release activate via upstream's draft-mtp speculator.
# -np must be 1 with MTP (parallel slots are not yet supported alongside MTP).
llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
-np 1 \
--jinja -ngl 99 -fa -c 131072 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
--presence-penalty 1.5
Run with Ollama:
ollama run hf.co/bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M
Download a single GGUF file:
hf download bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF \
Qwen3.6-27B-MTP-pi-tune-Q4_K_M.gguf \
--local-dir .
Download the whole repo:
hf download bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF --local-dir .
This release is compatible with the Qwen3.6 mmproj-F16.gguf sidecar for vision-language inference. A single mmproj file pairs with every quant in this release; the projector is architecturally tied to the base model's vision tower, not to the LM quant level, so download it once and reuse it.
The compatible mmproj can be downloaded from unsloth/Qwen3.6-27B-MTP-GGUF. The fine-tune in this release is language-only — the vision encoder weights have not been touched. Image / video understanding is therefore inherited unchanged from the upstream Qwen3.6-27B base model; this release does not claim to improve it, only to preserve it.
# Pull the LM weights from this repo
hf download bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF \
Qwen3.6-27B-MTP-pi-tune-Q4_K_M.gguf \
--local-dir .
# Pull the compatible mmproj sidecar
hf download unsloth/Qwen3.6-27B-MTP-GGUF \
mmproj-F16.gguf \
--local-dir .
# Launch llama-server with vision attached
llama-server -m ./Qwen3.6-27B-MTP-pi-tune-Q4_K_M.gguf \
--mmproj ./mmproj-F16.gguf \
--jinja -ngl 99 -fa -c 131072 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
--presence-penalty 1.5
For a quick text-and-image session without spinning up a server:
llama-mtmd-cli -m ./Qwen3.6-27B-MTP-pi-tune-Q4_K_M.gguf \
--mmproj ./mmproj-F16.gguf
llama-server exposes an OpenAI-compatible /v1/chat/completions endpoint, so any client written against the OpenAI SDK can point at it directly — no client changes required:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed",
)
resp = client.chat.completions.create(
model="Qwen3.6-27B-MTP-pi-tune",
messages=[
{"role": "system", "content": "You are a precise coding agent."},
{"role": "user", "content": "Write a Python function that merges overlapping intervals."},
],
)
print(resp.choices[0].message.content)
The same endpoint accepts tools=[...] for function calling and supports streaming via stream=True.
Tuned on real agent traces, not synthetic generations or distilled chat — every training row is the trail of an assistant that actually executed the task end-to-end through a PI-style harness, exported as Qwen-compatible ChatML rows with tool schemas and runtime prompts preserved.
High-level task coverage spanned:
Specific dataset names and training-row counts are intentionally omitted from this initial card.
MTP stands for Multi-Token Prediction. The model drafts likely future tokens and the runtime accepts them when they agree with the main decode path. On local agent work this matters because long reasoning, code generation, tool-call setup, and shell-oriented turns otherwise spend most of their wall time waiting on generation.
The numbers below describe the current local profile of the release. They are representative figures from internal runs against the PI harness — full benchmark publication is forthcoming and will replace these with task-success-rate tables in §7.
The agentic number is different from raw tokens/sec. It measures real task throughput across agent runs — including model generation, tool calls, shell commands, package installs, file I/O, and verifier-facing work.
<div style="display:grid;grid-template-columns:repeat(2,1fr);gap:14px;margin:14px 0 18px 0;"> <div style="border:1px solid #e2e8f0;border-left:4px solid #6d28d9;border-radius:10px;padding:16px 18px;background:#ffffff;"> <div style="font-size:10.5px;font-weight:700;color:#6d28d9;text-transform:uppercase;letter-spacing:0.1em;">Effective Output</div> <div style="font-size:26px;font-weight:800;color:#0f172a;margin-top:6px;line-height:1;">~33 <span style="font-size:13px;font-weight:600;color:#64748b;">tok/s</span></div> <div style="font-size:12px;color:#64748b;margin-top:6px;line-height:1.45;">Output tokens divided by full agent execution time.</div> </div> <div style="border:1px solid #e2e8f0;border-left:4px solid #0e7490;border-radius:10px;padding:16px 18px;background:#ffffff;"> <div style="font-size:10.5px;font-weight:700;color:#0e7490;text-transform:uppercase;letter-spacing:0.1em;">Effective Total</div> <div style="font-size:26px;font-weight:800;color:#0f172a;margin-top:6px;line-height:1;">~1.6k <span style="font-size:13px;font-weight:600;color:#64748b;">tok/s</span></div> <div style="font-size:12px;color:#64748b;margin-top:6px;line-height:1.45;">Input + output tokens counted by the harness, end-to-end.</div> </div> </div>Effective output throughput is computed as:
sum(output tokens) / sum(agent execution duration)
That makes it a more realistic agent-workflow number than plain decode speed — it includes time spent operating through the harness, not just time spent generating text.
The follow-up release will cover task-success rates across the high-level areas listed in §5: terminal/shell agent tasks, tool & function calling, multi-language code editing, repository issue resolution, and coding/API integration tasks.
[!IMPORTANT] Throughput figures above are from the local MTP-enabled run. Task success rates should be reported only from completed eval runs, not inferred from speed.
Released under the Apache 2.0 license, inherited from the upstream Qwen3.6-27B base model. You are free to use, modify, and redistribute the model and its derivatives subject to the terms of that license.
Thanks to the Qwen team for the Qwen3.6 base model and its MTP design, to the ggml-org / llama.cpp maintainers for native multi-token-prediction support in upstream, and to the broader open-source quantization tooling community whose work makes local-first inference of frontier models possible.
<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;border:1px solid #c7d2fe;border-radius:18px;overflow:hidden;background:#ffffff;margin:36px 0 8px 0;box-shadow:0 6px 20px -12px rgba(79,70,229,0.35);"> <div style="background:radial-gradient(circle at 10% 90%, rgba(45,212,191,0.45) 0%, rgba(45,212,191,0) 55%), linear-gradient(120deg,#0e7490 0%,#4338ca 100%);padding:26px 28px;color:white;"> <div style="font-size:11px;font-weight:800;letter-spacing:0.18em;text-transform:uppercase;color:#a5f3fc;margin-bottom:8px;">Built for local agent loops</div> <div style="font-size:20px;font-weight:800;color:white;line-height:1.3;">Speed where it matters · context where it counts · MTP at every quant.</div> <div style="display:flex;gap:18px;flex-wrap:wrap;margin-top:18px;"> <div> <div style="font-size:24px;font-weight:800;color:#ffffff;line-height:1;">128k</div> <div style="font-size:11px;color:#c7d2fe;margin-top:2px;letter-spacing:0.06em;">tested · 256k native · 1M max</div> </div> <div style="width:1px;background:rgba(255,255,255,0.18);"></div> <div> <div style="font-size:24px;font-weight:800;color:#ffffff;line-height:1;">~78%</div> <div style="font-size:11px;color:#c7d2fe;margin-top:2px;letter-spacing:0.06em;">MTP draft acceptance</div> </div> <div style="width:1px;background:rgba(255,255,255,0.18);"></div> <div> <div style="font-size:24px;font-weight:800;color:#ffffff;line-height:1;">~1.6k <span style="font-size:13px;font-weight:600;color:#a5f3fc;">tok/s</span></div> <div style="font-size:11px;color:#c7d2fe;margin-top:2px;letter-spacing:0.06em;">end-to-end agent throughput</div> </div> </div> </div> </div>