Kleiner-Coder-35B-vram13-MTP

🔬 Kleiner — Qwopus-Coder VRAM13 + Corpus-Tuned MTP

Black Mesa series · the coder, all-VRAM, with a finetuned multi-token head.

~13.5 GB all-VRAM GGUF of Jackrong's Qwopus-3.6-35B-A3B-Coder (agentic coding fine-tune of Qwen3.6-35B-A3B), with a corpus-finetuned MTP (nextn) head swapped in for self-speculative decoding under ik_llama.cpp (-mtp).

This is a separate release from the earlier native-head VRAM13 Qwopus build. Same backbone quant recipe; new blk.40 head trained on a held-out corpus (val loss ≈ 2.10, top-1 ≈ 62.2%), then exported into the GGUF.

TL;DR: coding model that fits dual mid-range GPUs (~13.5 GB weights), runs with verification-lossless MTP decode at ~144 t/s code / ~120 t/s prose on a 3060 Ti + 3080 desktop. Thinking-off by design — keep --reasoning off for agent loops.

---

File

Measured performance

Rig: RTX 3060 Ti 8 GB + RTX 3080 10 GB, DDR4, ik_llama.cpp, q4_0 KV, flash-attn, all layers on GPU, -mtp.

Speculative decoding verifies every draft token against the backbone, so output tokens match the non-MTP path; only throughput changes with acceptance rate (code usually accepts better than free prose).

Recipe

Backbone (VRAM13 / q2ex + imatrix)

Requantized from Q8 with a Qwopus-specific importance matrix (ik/TQ llama-quantize custom rules; nextn-aware — mainline can choke on the MTP layer):

• Experts blk 3–36 → Q2_K; edge experts 0–2 & 37–39 → Q3_K
• Attention + SSM projections → Q5_K
• Shared expert, output, embeddings → Q6_K
• Base ftype Q4_K so every custom rule fires

Principle: on small-active MoE, experts dominate size and tolerate 2-bit with imatrix; the every-token path stays higher precision.

MTP head (corpus-tuned)

The stock blk.40 nextn tensors were replaced (not grafted from another model) using a head trained offline against frozen backbone hidden states:

• Large MoE expert mats → Q4_0
• Attention / shared-expert / eh_proj → Q8_0
• Norms / router → F32, with Qwen3Next RMSNorm written as (1 + HF weight) (GGUF convention)
• ~0.50 GB head footprint

qwen35moe.block_count = 41, nextn_predict_layers = 1.

Training (MTP head only)

Only the nextn / MTP module is trained (~835M params). Backbone, embeddings, and lm_head stay frozen. The objective matches self-speculative decode: predict token t+2 from backbone hidden h_t and embed(t+1).

Rough scale: ~1.7M tokens is a small, domain-specific finetune — enough to pull the draft head toward real agent/coding traces on this backbone, not a general pretrain. Speedups still come from draft acceptance; verified tokens are always the backbone’s.

How to run

Requires ik_llama.cpp for reliable nextn load + -mtp. Mainline may load the file but will not drive the head the same way.

Recommended (long agent / tool sessions — 64K)

Flash-attn + MTP KV can OOM mid-session at deeper contexts on ~18 GB dual-GPU rigs. 64K is the practical ceiling for sustained agent work on this hardware class:

./llama-server \
  -m Qwopus3.6-35B-A3B-Coder-vram13-q2ex-imat-MTP.gguf \
  --jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
  --ctx-size 65536 --parallel 1 --n-gpu-layers 99 --ctx-checkpoints 8 \
  --tensor-split 44,56 --ubatch-size 512 \
  -mtp --ctx-size-draft 8192 \
  --reasoning off --reasoning-budget 0 \
  --no-mmap --threads 8 --no-warmup --port 8000

Short interactive / more headroom

You can try 128K with a smaller draft context and careful batch sizes; if you hit FA/KV OOMs under load, drop back to 64K or shrink --ctx-size-draft / --ubatch-size.

• Qwopus is a thinking-off design — keep --reasoning off (also avoids empty-content stalls in some clients).
• Single ≥16 GB GPU: drop --tensor-split.
• Drop -mtp if you only want the backbone (no speculative speedup).

Intended use & limitations

• Target: local agentic / tool-use coding with high decode speed on ~16–18 GB total VRAM.
• 2-bit experts are the quality floor vs larger mixed-q2k or Q8 builds; this file optimizes all-VRAM speed, not maximum fidelity.
• MTP acceptance (and thus speedup) varies by domain; code-like text is usually strongest.
• Inherits capabilities and biases of the Qwopus-Coder fine-tune. Head swap does not change verified next-token distribution of the backbone.

Related

• Parent coding fine-tune: Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF
• Mixed q2_K + imatrix (CPU offload, 256K): Kleiner-Coder-35B-mixed-q2k
• Base Qwen VRAM13 (non-coder): see other xero0000 VRAM13 uploads

Provenance

• Base: Qwen3.6-35B-A3B (Alibaba / Qwen, Apache-2.0)
• Coding fine-tune: Jackrong (Qwopus-3.6-35B-A3B-Coder)
• VRAM13 quant + imatrix, corpus MTP train/export, benches: xero0000, July 2026

Released under Apache-2.0 (same family as the base).