smollm2-135m-sql-drafter

SmolLM2-135M SQL Drafter

A draft model for speculative decoding, fine-tuned to predict what HuggingFaceTB/SmolLM2-360M-Instruct outputs on text-to-SQL prompts.

The point

A drafter's job is not to be good at SQL. It is to predict what the target model says next. So this was trained by on-policy distillation: SQL prompts were run through the 360M target, and this 135M model was fine-tuned on the target's own completions — not on the dataset's ground-truth SQL.

Train on ground truth and you get a model that writes good SQL and still gets rejected by the verifier.

Measured accept length

Target SmolLM2-360M-Instruct, 200 held-out text-to-SQL prompts. Accept length = tokens emitted per target forward pass. Ceiling is k+1.

The gain grows with k: at shallow depth the ceiling caps both drafters, but the deeper you speculate the more a generic drafter drifts on its own guesses while a target-matched one holds.

Use with vLLM

vllm serve HuggingFaceTB/SmolLM2-360M-Instruct \
  --speculative-config '{"method":"draft_model","model":"DEXX23/smollm2-135m-sql-drafter","num_speculative_tokens":7}'

Speculative decoding is lossless — the target verifies every token, so a draft model can only change speed, never output.

Training

b-mc2/sql-create-context prompts → completions generated by the target → 3 epochs, lr 1e-4, bf16. About 10 minutes on one GPU; fits a 4GB card.