Ornith-1.0-35B-MTP

Ornith-1.0-35B-MTP

Ornith-1.0-35B with a multi-token-prediction head grafted on, so it can run MTP speculative decoding in vLLM.

Ornith-1.0-35B ships no mtp.* tensors — verified, 0 of 31,666 — so --speculative-config.method=mtp has nothing to load. This checkpoint adds the 19 mtp.* tensors from its base model, Qwen/Qwen3.6-35B-A3B. Nothing was retrained, merged, quantized, or converted.

Both halves of that are checked rather than asserted:

• Ornith's weights are untouched. All 16 of its .safetensors shards have sha256 identical to the blobs in ornith-ai/Ornith-1.0-35B, so this repo's copies are bit-identical to upstream's. The only file added is model-mtp-00001-of-00001.safetensors.
• The grafted head is a byte-for-byte copy. All 19 mtp.* tensors match the Qwen base by per-tensor sha256, with identical shapes and dtypes.

Measured +38.7% on prose and +54.6% on code generation, at 70.3% draft acceptance.

The two flags that matter:

vllm serve crucible-labs/Ornith-1.0-35B-MTP \
  --speculative-config.method=mtp \
  --speculative-config.num_speculative_tokens=2

Everything below was measured with the full command that follows, on the hardware described under Results. Most of these flags are ROCm- and deployment-specific rather than required for MTP:

vllm serve <model> \
  --tensor-parallel-size 1 --dtype auto --trust-remote-code \
  --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.90 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN --enable-prefix-caching \
  --speculative-config.method=mtp --speculative-config.num_speculative_tokens=2 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --chat-template chat_template.jinja

Note the flag spelling: the dotted form is used because the JSON form of --speculative-config loses its quotes when passed through a systemd Environment= line. Both work on the command line.

Prior art, and what is different here

skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF did this graft first, in the GGUF world — with a surgery script, donor-head-only files, and reported acceptance of 74.3% (Q6_K) to 79.5% (Q4_K_M) at --spec-draft-n-max 4. If you serve with llama.cpp, use theirs, not this.

This repo is the counterpart for the other stack:

• safetensors at BF16 for vLLM, not quantized GGUF for llama.cpp — discussion #2 asks for exactly this and had no answer.
• Weights are unquantized and bit-identical to upstream, so nothing here is entangled with a quantization choice.
• Adds a drift measurement justifying why the graft should work at all, and tool-calling validation (400 problems, zero malformed calls), which matters because speculation is only output-preserving in exact arithmetic.

It also makes the checkpoint self-consistent: Ornith's config.json declares mtp_num_hidden_layers: 1 while shipping no mtp.* tensors, and that mismatch breaks some quantization toolchains outright (discussion #15). Here the config and the weights finally agree.

If you already have Ornith-1.0-35B downloaded, you do not need this repo — you can build it locally in seconds with graft_mtp_head.py and save the 67 GB transfer. This repo exists for convenience.

Why this works

The MTP head predicts token t+2 from the main model's hidden state. Ornith is a light post-train of Qwen3.6-35B-A3B, so that hidden space barely moved and the base's head still applies. Measured drift between Ornith and its base:

• median weight drift 2.46%
• cosine similarity ≥ 0.998 on every sampled tensor
• final norm bit-identical

The result is self-verifying: a head that no longer matched the hidden space would produce drafts that get rejected, showing up immediately as low acceptance and a slower model. Instead acceptance is 70.3%, which is high enough to pay for the verify batch several times over.

(For loose context, the base model running its own MTP head measured 79% acceptance at n=2 in a separate test — but that was on llama.cpp rather than vLLM, so the two are not a controlled comparison and the difference should not be read as the cost of the post-train.)

Results

Batch 1, temperature 0, warmup run discarded. All arms used identical serving flags apart from the speculative config; the control arm ran with speculation disabled. Single Strix Halo (gfx1151 / Radeon 8060S, 124 GB unified memory), ROCm 7.14, vLLM 0.27.1, TP=1.

Use num_speculative_tokens=2 unless you generate only code. n=4 drafts more but acceptance collapses to 44.3%, and prose ends up 22% worse than n=2. More speculation is not better speculation.

Why it beats the bandwidth ceiling

This hardware reads 5.488 GiB per token at a measured 224.2 GiB/s, which caps single-token decode at 40.9 tok/s no matter how well the kernels are tuned. At n=2, code generation measured 42.51 tok/s — past the ceiling.

That is the point of speculation rather than an error: when 2.4 tokens come out of one forward pass, they share a single weight read. Speculative decoding doesn't make memory faster — it amortizes one read across several tokens, which is the only way past a per-token bandwidth ceiling.

Forward cost is linear in verify rows (n+1 rows per forward). Fitting the three measured points above gives ~12.9 ms per row + ~23.3 ms fixed, which reproduces all three within 0.5 ms. Roughly 7 ms of that fixed term is the draft head reading its own 1.573 GiB at the measured bandwidth. The linearity is why the sweet spot is small: each extra speculative token costs a full verify row whether or not it is accepted, so it only pays while acceptance stays high.

What it costs, and what will bite you

• KV cache capacity −21.8%. The draft head's own KV state has to live somewhere. The relative cost is the portable number; the absolute capacity depends entirely on your hardware and serving flags. On the configuration above it was 1,028,411 → 804,046 tokens, measured at --max-model-len 262144 --gpu-memory-utilization 0.90 --max-num-seqs 64 on a 124 GB unified-memory device — expect a completely different absolute figure on a discrete GPU. On long-context agent work this trade may matter more than the decode win.
• Keep num_speculative_tokens ≤ 4 on ROCm. The skinny-GEMM guard is 0 < n <= 5 and the verify batch is n + 1, so at n=5 every dense GEMM falls to a slow path. It is a cliff, not a gradient.
• Requires mamba_cache_mode='align' — the MTP class raises NotImplementedError on 'all'. With prefix caching enabled this was already the mode in use here, so it cost nothing, but check yours.
• min_p and logit_bias silently stop working under speculative decoding in vLLM. It logs this once at startup and never again. If your clients send them, they are being ignored.
• Output is not bit-identical to non-speculative decoding. The verify step runs a batched forward, so the target's own logits come from differently-tiled GEMMs than at batch 1 and the argmax flips at near-ties. In greedy runs, output matched the non-speculative reference for ~375 characters and then diverged onto an equally plausible continuation.

Validation

Speculation being output-preserving only under exact arithmetic is a real risk for structured output — n-gram speculation on this same model produced a malformed tool call, with the function-name field containing get_weather(location='Boston') and empty arguments. So tool calling was validated on a real benchmark rather than a smoke test:

• 400 BFCL-simple tool-calling problems, zero malformed tool calls.
• Run twice: outcome-identical — same failing IDs, same counts, 3334.1 s vs 3335.8 s.

A benchmarking trap worth knowing about

In that comparison MTP first appeared to break 10 problems, 8 of them "no tool call emitted" — which looks exactly like speculation damaging structured output.

It was the harness's token cap. A <tool_call> block truncated mid-stream parses to no tool call and empty content, which is byte-identical to a model that emitted nothing. Replaying those problems at double the budget: 8 of 10 had hit finish_reason: "length", and they needed 1082–1461 tokens where the cap allowed 1024.

The general mechanism: speculation changes the generation trajectory, so it changes output length, so a token cap that was previously sufficient can start clipping. The regression then gets blamed on speculation, because that is the variable you changed. Record finish_reason per sample in your eval harness — without it, a truncated tool call and a silent model are indistinguishable after the fact.

How it was built

1. Locate the 19 mtp.* tensors in Qwen/Qwen3.6-35B-A3B and copy their bytes into a new shard.
2. Hard-link Ornith's 16 existing shards (so the working directory costs 1.6 GiB locally, not 67 GB).
3. Rewrite model.safetensors.index.json to map the 19 new keys to the new shard.

Tensor counts: Ornith 31,666 + 19 = 31,685. The grafted tensors keep the base's 3D fused expert layout (mtp.layers.0.mlp.experts.gate_up_proj as [256, 1024, 2048]) even though Ornith stores its own experts 2D per-expert. That mismatch is fine and is the detail that makes this possible: the MTP module is loaded by a different model class than the main model, so the two layouts never have to agree.

No config surgery was needed — mtp_num_hidden_layers: 1 was already present in Ornith's config, and vLLM auto-derives architectures: ["Qwen3_5MoeMTP"].

Script: graft_mtp_head.py. It works for any base/finetune pair where the base ships an MTP head and the finetune is a light post-train of it.

Limits, stated honestly

• One model, one hardware configuration. Measured on gfx1151; acceptance should carry across hardware but throughput deltas will not.
• The control arm already included a separate ROCm dispatch fix (vllm#52631). The +38.7% is MTP's contribution on top of that, not the two combined.
• Prefill and TTFT under MTP were not isolated. The draft head prefills too, so expect a small cost; that is reasoning, not measurement.
• Optimal n is workload-dependent — prose and code disagreed here and will disagree differently for you.
• Quality was validated on tool calling specifically, since that was the identified risk. It is not a full re-benchmark of Ornith's capabilities, which are unchanged by construction: its weights are bit-identical to upstream.

Licensing and attribution

This checkpoint combines material under two licenses. Both are permissive and compatible; neither is modified by this release.

The Apache-2.0 license text is included as LICENSE.qwen3.6-apache-2.0, per its section 4(a). The mtp.* tensors are redistributed unmodified; the only change to the Ornith checkpoint is additive (19 new tensor entries in the index).

Ornith-1.0-35B declares MIT in its model card metadata. Its repository does not currently ship a LICENSE file, so there is no upstream copyright notice to reproduce here; attribution is given by direct link above. If the Ornith authors would like specific notice text carried in this repo, open a discussion and it will be added.

Neither upstream project endorses this repository.