Holodeck-Lounge-Regressed-MTP

Holodeck-Lounge — MTP Head Fine-Tune (Learning Experiment)

TL;DR — a documented negative result. This model is the MTP-restored Holodeck-Lounge (a working native single-layer DeepSeek-style MTP head: the merge already carried 14 of the head's 15 layers, and we restored it by sourcing only the missing mtp.fc.weight tensor from Jackrong/Qwopus3.5-9B-Coder, with the froggeric v22.1 chat template baked in) — with only that MTP head re-trained in isolation (backbone frozen) on self-distilled data. The fine-tune regressed MTP acceptance versus the donor head — 49.75% vs 58.55% under rejection-sampling acceptance. A failure consistent with what the literature predicts. It is shared as a learning artifact, not a production model.

📚 Directly related work

This experiment is a direct, small-scale test of claims from these papers:

⚠️ REGRESSION WARNING — research artifact, not for production

• This model's MTP head regressed versus the donor head — 59.0% / 58.55% at our standard eval (reasoning off). The original reaches 60–69% only with reasoning enabled; this fine-tune does not, in any configuration.
• The native MTP head here has lower acceptance than the donor head, offers no quality or speed benefit, and the shared backbone is untouched.
• Use the original model (Wiself/Holodeck-Lounge-MTP) for any real work.
• This model exists to document why such fine-tunes fail — and how to avoid it.

What we did

• Base: Wiself/Holodeck-Lounge-MTP — Holodeck-Lounge backbone (≈9.6B params, qwen3_5, 32 layers, hidden 4096) + transplanted single-layer MTP head (3 draft tokens) + froggeric v22.1 chat template.
• Fine-tuned only the MTP head (~300M params), backbone fully frozen, for 1 epoch on 5,000 self-distilled samples (greedy generations from the model itself).
• Loss: lk_hybrid (adaptive λ·KL + (1−λ)·TV, internal η=3, fused Triton impl).
• Signal caveat: the self-distilled targets were greedy hard labels, not teacher logits — 2602.06019 and MTP-D both distill teacher distributions (KL), which we did not.
• LR 2e-4, cosine schedule, 100 warmup steps, seq len 1536, RTX 5090 (32 GB), via vllm-project/speculators scripts/train.py.

Results — trained vs donor

Eval: 30 prompts, temperature 0.7, max 256 tokens, concurrency 8. llama.cpp numbers are greedy target-only acceptance (reasoning off); vLLM RS acceptance from vllm:spec_decode_num_accepted_tokens / _num_draft_tokens.

Read: the trained head loses roughly 10 points on aggregate acceptance; the deepest draft position (pos2) collapses from 41% to 30%.

Eval caveat: n=30 is a small sample — treat the 3-decimal precision as indicative; confidence intervals are wide.

Why it regressed — what the literature predicts

1. Learning rate too hot. On MTP for Efficient LLM Inference reports LR 6e-6 at 2.8B and LR shrinking with model size; a ~9B head-only fine-tune extrapolates to ~1e-6–3e-6. Our 2e-4 is 1–2 orders of magnitude hotter, destroying the head's pre-existing distribution alignment.
2. Corpus distribution shift. AdaMTP found that on Qwen2.5-7B all fine-tuning paradigms — including plain NTP — regressed below the base model, attributed to the fine-tune corpus, not the MTP paradigm. Our 5k self-distilled corpus diverges from the model's pretraining/eval distribution the same way.
3. Heads-only ceiling. 2502.09419 shows hidden layers are NTP-specialized; head-only adaptation has a hard ceiling and cannot match a head co-trained with the backbone.
4. We deviated from the proven recipes. Self-distillation (2602.06019) uses teacher likelihoods at LR 1e-5; MTP-D uses KL-to-main-head with stop-grad; FastMTP uses weighted CE at LR 5e-5 for 3 epochs — on ~389K samples, ~80× more than ours. We used a hot lk_hybrid loss (λ·KL + (1−λ)·TV) on self-generated tokens for 1 epoch — a maximally hostile combination.
5. Target entropy, not just weights. Bebop (Qwen team) shows MTP acceptance collapse during post-training is dominated by the target model's entropy, and prescribes a one-time TV-loss adaptation on a small corpus, then freeze. A hot LR over greedy hard targets is the regime their TV loss is designed to avoid.

Caveat — the experiment is confounded. With LR 2e-4 + greedy hard targets + 5k samples + 1 epoch, we cannot isolate which factor dominated. This documents combined failure modes, not a clean ablation.

How to use

Honestly: don't. This head is strictly worse than the donor's. The backbone text quality is unchanged (the MTP head only affects speculative decoding), so this is safe, but it is strictly slower to verify than the original. Use the original at Wiself/Holodeck-Lounge-MTP for MTP acceleration — it delivers a 60–69% acceptance rate (65+% typical) with the recommended settings (see its Real-world usage section), which this fine-tune does not.

Training details

Provenance & credits

AI was used to draft this report.