Qwen3.x-27B-mlx-mtp-graft

MTP graft — restoring Multi-Token-Prediction heads to MLX quantizations that dropped them

Most MLX quantizations of Qwen3.5/3.6/3.8 keep mtp_num_hidden_layers: 1 in config.json while silently dropping the MTP weights. The checkpoint loads, generates correctly, and gives you no reason to suspect anything — it just cannot do multi-token prediction, and nothing tells you why.

This repo contains the MTP head only (314 MB), plus a tool to graft it onto a recipient checkpoint. No base weights are redistributed.

Measured on an Apple M5 Max, 128 GB, macOS 26.4.1.

What's in here, and what it applies to

Not claimed to work on Qwen3.5-27B or Qwen3.6-27B — not tested. Nothing here was trained; the head is Qwen's own (see Provenance).

Result

Recipient: Qwen3.8-27B abliterated with Heretic (trohrbaugh/Qwen3.8-27B-heretic-ara, MLX affine 4-bit g64), which ships with 0 MTP tensors. Served through oMLX, greedy, thinking off, A/B/A interleaved with cooldowns, 3 rounds:

Acceptance at depth 1 is 93.7% on code, 53% on Chinese prose, from oMLX's own per-request log. tok/cycle 3.53 and 1.88 respectively.

How the head is packaged matters more than any of this, and it fails silently. Read the next two sections before drawing conclusions from a benchmark.

Packaging, not weights: how the same head scores 1.6% or 97%

The head that gives 93.7% acceptance under oMLX gave 1.56% the first time it was handed to MTPLX's runtime. It did not error. It drafted noise, every draft was rejected, and MTP quietly became slower than not using MTP at all.

It was not the contract parameters. All four combinations were swept:

It was the packaging. The two runtimes want the same head in incompatible containers:

Given a 4-bit head, MTPLX reads the U32 packed words as if they were the weight matrix. Shapes are plausible, nothing raises, acceptance goes to zero. This is the worst failure mode in the whole exercise: silent degradation to "MTP does not help here", which reads exactly like a negative result.

to_mtplx.py in this repo dequantises the head to fp16, renames it, drops it out of the index and writes a mtplx_runtime.json. Same artifact, after conversion:

D1 acceptance 96.79%   (was 1.56%)
D3 acceptance 98.0 / 90.9 / 84.9%

Numbers, and a retraction

An earlier revision of this README claimed 3.79× for MTPLX against oMLX's 2.11×. That number is withdrawn. It came from a tune run that generated 200 tokens while the comparison runs generated 512 — short runs decode at a shorter context and flatter the multiplier. Re-run under identical settings, the same artifact gives 2.68×.

MTPLX 2.9.1, tune --depths 1,2,3, 512 tokens, M5 Max 128 GB, interleaved A/B/C then C/B/A, best depth per run:

Three things worth taking from this table:

Run-to-run drift is large and directional. Five consecutive tune runs with 5 s settling gave AR 31.0 → 27.4 → 24.9 tok/s — a 20% slide from thermals alone, enough to manufacture or destroy any conclusion at this effect size. Interleave, repeat, and report the spread; a single tune is not a measurement.

Higher acceptance did not mean higher throughput. The official fp16 head accepts more than the grafted one (98.92% vs 96.79%; 100% at the first draft position at depth 3) and is consistently slower. Acceptance is a model property, throughput is a systems property, and optimising the first does not automatically move the second. The mechanism here is not isolated — the two artifacts differ only in the contents of mtp.safetensors, identical shapes and dtype, so a 12–15% throughput gap should not exist. Reported as measured and unexplained rather than rationalised.

The gap to a natively-forged artifact is real but modest. 58.8 vs 71.7 tok/s at near-equal acceptance. Grafting recovers most, not all, of what a purpose-built MTP release gives you.

On draft-head precision

An earlier revision concluded that fp16 heads are a bad trade, from 252 ms/draft (fp16) vs 27 ms (4-bit) measured inside oMLX. In MTPLX the same fp16 head costs ~3.4 ms per call. That comparison was confounded by the runtime and is void. What the table above says instead: after conversion both are fp16 anyway, and the fp16 head that accepts best is not the one that runs fastest.

On depth

oMLX runs an adaptive controller (cap 8, not 3 — an earlier claim of 3 was the controller's choice, not its limit) scoring (1 + p1 + p1·p2 + …) / t_est(d) from live measurements. MTPLX's tune picks per artifact and disagrees with itself across reps: depth 3 won 3 of 5 runs for the graft, depth 2 won 3 of 5 for the reference, and the margin between them is inside the drift band above. Tune on your own machine, and do not trust a single tune.

Sparse MoE: no gain under oMLX

Grafted onto a qwen3_5_moe 35B-A3B recipient (donor: mlx-community/Qwen3.6-35B-A3B-MTP-4bit, 46 tensors), oMLX reports Lightning MTP (model_type=qwen3_5_moe, active) and 59% acceptance — and buys nothing: 102–117 tok/s ungrafted, 101–108 grafted. The controller detects it and stops:

MTP[1] finish=parked  tok/cycle=1.29  accept=25.8%  depth[d1=8/31, d2=0/0, d3=0/0]

oMLX's source names the case: "MoE expert loads scale with verify tokens … the whole depth menu can be worse than standard decoding." The MoE's single-token step is already 9.5 ms against the dense model's 33.8 ms; there is little to amortise.

Counter-evidence, and it is not weak: EryriLabs grafts a head onto the same architecture and reports +29% under llama.cpp on an RTX 3090, at comparable acceptance. Given the MTPLX result above, the likeliest explanation is again the runtime, not the architecture. Measure your stack; do not take the MoE result here as general.

Output fidelity

Verify-then-accept means the head cannot make the model say something the base model rejects. But batched K-token verification and single-token decoding take different floating-point paths, so near-ties flip at temperature 0.

Fixed seed, temperature=0, top_p=1, top_k=0:

before: …因為它假設近期被使用過的資料[[，未來被再次使用的機率較高。]]
after : …因為它假設近期被使用過的資料[[在未來被再次使用的機率較高。]]

Both fluent, neither better. Low-entropy text has no near-ties and reproduces exactly. If you need bit-reproducible output, do not enable MTP.

Known incompatibility: mlx_lm breaks on MTP tensors

A checkpoint containing MTP tensors produces garbage under mlx_lm.load, silently. Controlled test, same checkpoint, only the 46 MTP tensors differing:

BigBang-v1-heretic-MLX-4bit      (MTP = 0)   -> normal Chinese output
BigBang-v1-heretic-MLX-4bit-mtp  (MTP = 46)  -> ";av L WL two one lya"**开始中j位ode理ments谷雨谷…"

• oMLX / MTPLX — work, and are the only stacks that use the head.
• mlx_vlm — works (ignores it).
• mlx_lm — broken. Keep an ungrafted copy if you need this path.

This cost me a wrong diagnosis: the same garbage appeared on a VL checkpoint that had both vision and MTP tensors, and I blamed vision. It is the MTP tensors.

Handing an oMLX graft to MTPLX (and back)

Going the wrong way at least fails loudly. oMLX given an MTPLX artifact:

VLM load failed: Received 15 parameters not in model: mtp.*
LLM fallback also failed: Lightning MTP is enabled ... the converted weights are
missing the mtp.* tensors. Default mlx-lm converters strip them; you need a
converter that preserves MTP.

(oMLX's author documents this exact failure mode in the error text.) Note also that oMLX caches load failures — Skipping load retry ... after cached failure — so a fixed artifact needs a restart, not a retry.

graft_mtp.py normalises mtp.*, bare standalone heads and already-namespaced language_model.mtp.* into the language_model.mtp.* layout oMLX wants. to_mtplx.py goes the other way. The direction that fails silently is the one that matters: MTPLX given a quantised head (see above).

Usage

python graft_mtp.py \
  --recipient /path/to/model-without-mtp \
  --donor     /path/to/donor-or-standalone-head \
  --out       /path/to/model-with-mtp

Recipient weights are hard-linked, not copied; the graft costs only the MTP shard. The original directory is never modified. The tool refuses mismatched hidden_size, head counts, vocab_size, num_experts, moe_intermediate_size, or a different base quantization scheme (--force overrides).

In oMLX: set mtp_enabled and unload/reload the model. oMLX reads model settings at load time only — editing model_settings.json while the model is resident does nothing (verified twice).

For MTPLX, convert first — the oMLX layout does not work there, and does not say so:

python to_mtplx.py /path/to/model-with-mtp /path/to/model-mtplx
mtplx inspect --model /path/to/model-mtplx     # want: tier: verified, can_run: true
mtplx tune    --model /path/to/model-mtplx --depths 1,2,3 --retune

inspect is the cheap gate: runtime_contract: false means no mtplx_runtime.json, and mtp_tensors_present: 29 instead of 15 means the head is still quantised — 29 counts scales and biases as tensors. Both states still report can_run: true.

Verifying a graft

Three checks, cheapest first.

1. Are the tensors there? Check weights, not config — the config lies. Of eight Qwen3.5/3.6/3.8 MLX checkpoints on hand, four declared mtp_num_hidden_layers: 1 with zero MTP tensors present.

wm = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"]
print(sum(1 for k in wm if "mtp" in k.lower() or "nextn" in k.lower()))

2. Are they trained? Counting is not enough — a head can be correctly shaped random initialisation. EryriLabs reported this on 2026-08-20 for Ornith-1.5-35B-A3B: 785 mtp.* tensors at std = 0.0200 with Gaussian kurtosis 3.0 — literally initializer_range — accepting ~13% of drafts. Every tensor-count check passes. (Fixed upstream on 2026-08-23; sampling six of those tensors by HTTP range request now gives norm means 0.74–0.87 and projection kurtosis 5.7–10.4, and they match EryriLabs' grafted head to within 1 bf16 ULP — upstream adopted the same fix.)

Two tells, no inference needed: norm weights sit at mean ≈1 or above when trained, at the initializer range when not; projections are heavy-tailed when trained, Gaussian (≈3.0) with identical std across every layer when not.

The head in this repo: norm means 1.036 / 1.206 / 1.791 / 1.779 / 2.252; projection kurtosis 8.10 / 3.71 / 14.65, std 0.0159 / 0.0127 / 0.0198. Trained, and corroborated by the 93.7% acceptance above.

3. Is the wiring right? The head consumes concat(embedding, hidden), and there are choices: which hidden (pre- or post-norm), concat order, and local vs global position indexing. Get one wrong and acceptance collapses without an error. MTPLX exposes them as --base-hidden-variant, --concat-order, and its artifacts declare an explicit mtp_contract.

Swept here, all four combinations, on the grafted artifact: see the table in §Packaging. Donor and recipient share a base, so the defaults were right — post_norm / embedding_hidden / local. Sweep anyway when grafting across less related checkpoints: it is four cheap runs, and a wrong contract is indistinguishable from a bad head. Note the corollary the sweep taught: a collapse that survives every contract setting is not a contract problem. Look at the container before you look at the wiring.

Provenance

The head is Qwen's own:

1. Qwen/Qwen3.8-27B — original MTP heads
2. tozp/Qwen3.8-27B-OBLITERATED-V2-mlx-4bit — MLX affine 4-bit g64; its abliteration_metadata.json records "Restore MTP + vision tensors from stock", so the heads it carries are unmodified
3. this repo — those 29 tensors, extracted unchanged

29 tensors, 314.3 MB, sha256[:16] = 505827ce17922d2d.

For qwen3_5_moe (hidden 2048, 256 experts) the donor is mlx-community/Qwen3.6-35B-A3B-MTP-4bit — same tool, 46 tensors, grafts cleanly. Read the MoE section first.

Limitations

• One machine (M5 Max), one MLX version. Both runtimes measured; two more exist (llama.cpp, vLLM) and are not.
• Prompts are mtplx tune's own English/code set plus a small hand set. CJK is measured only under oMLX, where acceptance is much lower (53% vs 93.7%) — the MTPLX numbers here should not be read as CJK performance.
• 512-token generations. No long-context, no batch >1, no concurrency.
• The graft-vs-official-head throughput gap (§Numbers) is measured but not explained.
• Grafting a stock head onto an abliterated model is not obviously sound in theory. Empirically: the recipient's KL to stock is 0.0535, while a checkpoint shipping this head natively sits at 0.9624 — the recipient is ~18× closer to stock than a model where the head demonstrably works. And the failure mode is bounded: a mismatched head lowers acceptance, it cannot corrupt output.
• Not tested with vision inputs on VL recipients, under either runtime.

Credits

• Qwen — the MTP heads
• tozp — the 4-bit quantization they came from
• Heretic / trohrbaugh — the recipient
• oMLX — Lightning MTP
• MTPLX — the runtime that showed how much of this is execution rather than weights, and the contract vocabulary
• EryriLabs — the random-init diagnostic and the llama.cpp MoE counter-result