Ornith-1.0-35B-MTP-head-v3-GGUF

Ornith-1.0-35B — retrained MTP draft heads

Two drop-in replacements for Ornith's MTP (nextn) speculative-decoding head, packaged as small donor GGUFs. You graft one onto whatever Ornith quant you already have instead of downloading another 36 GB.

Neither one required the original safetensors. Hidden states were captured straight out of the Q8_0 GGUF, the head was pulled from that same file as a starting point, trained in PyTorch, and written back in. If all you have is a GGUF, you can still improve its draft head.

The short version

We spent a day and a half trying to make speculative decoding faster on a pair of MI50s. Almost everything we tried did nothing. What worked was changing one line: what the loss compares against.

What we saw

The stock head drafts badly once conversations get long. On our box acceptance sat around 74-75% on short prompts and fell off past 8k, and every rejected draft is wasted work, so that comes straight off your tokens per second.

First attempt — the v3 file here — was the obvious one. Capture hidden states from the quantised model, train the head on them, patch it back. Cross-entropy against the next-next token in the corpus, which is how these heads normally get trained. It helped at long context, 32k acceptance went from 62% to 74%, and did nothing anywhere else.

Then we tried what you'd expect to try next. More data: 1.4M tokens of fresh captures with full 32k conversations, which was the whole reason we bothered fixing the capture tool. That bought nothing and was slightly worse at 32k. Two more epochs on the existing data: better validation score, worse in the actual A/B, which is its own lesson. Other drafters, DFlash and five flavours of ngram: all lost. Every serving knob we could find — draft depth, p-min, KV cache quantisation, kernel selection, split modes — no change or worse.

What actually changed

While working out why the validation number kept disagreeing with the real benchmark, the mismatch got obvious. We were training the head to predict what the text said. Acceptance doesn't check that. It checks whether the draft matches what the target model itself would have produced. Those differ any time the model would have written something other than what the corpus author wrote, which is often.

No new data needed. The captures already contain the answer: result_norm[i+1] is the target's own final hidden state for position i+2, so pushing it through the LM head gives you exactly what the big model would say next.

teacher = (h_next.to(torch.bfloat16) @ lm_head.t()).argmax(-1)
loss    = F.cross_entropy(draft_logits, teacher)   # was: the corpus token

That's the whole change, about ten lines with the dataloader edit.

What we expected, and what we got

We expected more than we got, because the first numbers we saw were wrong. An early run reported roughly +6.5 points. That turned out to be contaminated: two runs had been split differently, so most of one run's validation set had sat in the other's training data. Measured again on a properly held-out set it was +2.2. Good, just not the jump it looked like.

On our hardware, against conversations shaped like what we actually serve — a Discord bot with a 1.5k-token system prompt and tool calls:

On generic technical prose for comparison: 75/75/74% became 76/78/74% at 512/8k/32k.

In wall-clock terms on two MI50s with Q8_0 split across both cards, that's about 99 tokens/s at 3k of context and 86 at 30k.

So a couple of points at short context, a decent chunk in the middle, less than we hoped at the very long end. Still the biggest single win we found, and it cost one afternoon of GPU time against days spent on things that didn't work.

One more negative result

A follow-up run stacked 2.92M tokens of new data on top of this, with full 32k attention windows. On the author's held-out set it scored +0.5. On our benchmark it came out 1.3 points worse, losing on three of four conversation lengths.

Both measurements are probably right and just measuring different things — their held-out data is long self-generated prose, ours is chat with tool calls. Which is the actual takeaway: once the objective was fixed, more data stopped transferring. Benchmark on traffic that looks like yours, and don't trust a validation number that isn't measuring acceptance.

Using it

Graft onto a base Ornith GGUF that has no MTP block. Target first, donor second:

python3 gguf_mtp_graft.py \
    ornith-1.0-35b-<your-quant>.gguf \
    ornith-1.0-35b-MTP-head-v31-distill-Q8_0.gguf \
    ornith-1.0-35b-<your-quant>-MTP.gguf

It appends the 20 blk.40.* tensors and bumps block_count 40 → 41 with nextn_predict_layers = 1. The head is Q8_0 and doesn't care what your base quant is.

Then:

llama-server -m ornith-1.0-35b-<your-quant>-MTP.gguf \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --gpu-layers 999 -c 262144

--spec-draft-n-max 2 was best on our hardware. Drafting deeper only paid off at mid-length context and hurt everywhere else. --spec-draft-p-min works but never bought us any throughput.

Things that will bite you

Benchmark on more than one prompt. Acceptance depends enormously on what the text looks like. Same model, five different 32k windows of the same corpus: 43%, 84%, 59%, 71%, 86%. A single greedy prompt reproduces to the token every run, which makes it feel reliable while it's telling you about one sample. We chased a long-context bug that didn't exist because of this.

Partial cache reuse will crash a hybrid model. Ornith has delta-net recurrent layers. llama.cpp reuses a slot when a new prompt partly matches the cached one and drops the divergent tail, and dropping any of it corrupts recurrent state and hard-faults the GPU. We saw a crash where the server had kept 97.6% of the cache. --slot-prompt-similarity doesn't save you, it gates on similarity rather than on whether a removal happens. Erase the slot instead, and don't pre-warm slots with a saved prefix — we ran a sidecar doing exactly that and it was quietly crashing our server for a day.

Validation accuracy is not acceptance. One of our checkpoints scored 92.5% against 89.7% and lost every live comparison. If the validation set comes from the same corpus you trained on, it rewards memorisation.

Credit

The distillation head was trained by a collaborator on a DGX Spark (GB10), starting from our v3 checkpoint and our captures. They also caught and corrected their own inflated first number, which is the main reason the figures above are trustworthy. The capture-tool fix that made long chunks possible got found independently at both ends.

Base model is Ornith-1.0-35B by deepreinforce-ai, MIT. The gfx906 Docker images that make any of this work on decade-old datacentre cards are from mixa3607/ML-gfx906.

Full pipeline, scripts and the longer write-up: https://github.com/Dreadnouhgt/ornith-mtp-gfx906