danish-lm-400m-sft-v5

danish-lm-400m-sft-v5

400M-param Danish LM, SFT'd on a 9-source mix. Successor to danish-lm-400m-sft-v4 — same base, same optimizer, only change is upgrading the instruction-following dataset from IF-v1 → IF-v2 (150 render variants, 5-way combos, held-out eval).

Training

• Base: jensjepsen/danish-lm-400m-base-ckpt310k
• Data: 1.088M rows across 9 sources (word-problems v2, metamath-gsm, algebra v5, arith-chain v1, wiki-grounded v3, text-to-question v2, sciq, gsm8k, instruction-following v2). No alpaca, no dolly.
• Recipe: 2 epochs, batch 32, LR 5e-5 cosine_with_min_lr, 200 warmup, packed 512-length, no morpheme preprocessing, completion-only loss.
• Final eval loss: 0.328 (from 0.416 → 0.328 across 8 checkpoints).

Evaluation (final ckpt)

v4 → v5 deltas (same evaluator, n=200):

• IF-v2 held-out: 6.0% → 41.0% (+35pt, ≈ 7× better)
• citizen-MC: 52% → 55% (+3pt)

Known limitations

• Generative/MC gap. Model discriminates well on MC (Cit 55%, SciQ 43%) but struggles to emit the correct answer as text (Cit-gen 15%, SciQ-open 10%, SciQ letter-emit at random chance). Same failure pattern as v4.
• GSM8K single-shot is weak (~20% pass@1). Format is learned; reasoning is the bottleneck. Best addressed with a GRPO pass.
• title_wrapped (<<...>>) format not learned — 0% pass on IF eval. Rare-syntax learning failure; unrelated to the entire_in_quotes issue.
• entire_in_quotes not trained — the IF-v2 generator had a bug that kept this constraint out of every training row (fixed in generator; will land in IF-v3 → SFT-v6).

Prompt format

<|user|> {question} <|assistant|> {answer} <|end|>