EuroLLM-1.7B-Instruct-LoRA-draft-pilot

EuroLLM-1.7B-Instruct LoRA SFT Alignment (Draft Pilot)

This is a LoRA adapter trained on utter-project/EuroLLM-1.7B-Instruct using the mix-v4 SFT dataset. It was built as a research pilot to serve as an aligned draft model for AMALIA-9B speculative decoding (DSpark).

SFT Training Details

• Base Model: utter-project/EuroLLM-1.7B-Instruct (quantized to 8-bit MLX)
• Dataset: datagen/mix-v4 (200 refusals, ~242 SFT real QA, ~313 arithmetic, 78 format)
• Iterations: 200 (batch size 2)
• Learning Rate: 1e-5
• Optimizer: AdamW

SFT Performance Metrics (Harness)

---

Conclusion: The Speculative Decoding Paradox

While SFT successfully aligned the draft model to the formatting and honesty rules of the target model (raising the pass rate from 7.7% to 47.3%), it did not yield the expected 2x-3x speedup during speculative decoding.

Benchmarks (Target: AMALIA-9B-8bit)

• Physics Prompt: 21.96 tok/s (no draft) vs. 23.24 tok/s (with draft) → 1.06x speedup
• Refusal Prompt: 20.69 tok/s (no draft) vs. 21.54 tok/s (with draft) → 1.04x speedup
• Arithmetic Prompt: 30.05 tok/s (no draft) vs. 12.82 tok/s (with draft) → 0.43x slowdown (57% hit)

Why?

1. Exact Match Requirement: Speculative decoding requires exact token-by-token matching. Standard SFT teaches the model the rules, but doesn't align its phrasing and style to match the target model's output distribution. Even when both models output correct/honest responses, slight differences in phrasing cause token rejection and eliminate speedups.
2. Arithmetic Mismatch: Because the draft model has a low (7%) arithmetic accuracy, the target model rejects almost all of its proposed tokens on arithmetic tasks, leading to a severe slowdown due to drafting overhead.

Next Step

To resolve this, we must run On-Policy Distillation—training the 1.7B draft model on the target model's exact text completions (AMALIA-9B's outputs) instead of the generic ground-truth training set.