Qwen3.5-9B-EU-SFT-GRPO

Qwen3.5-9B-EU-SFT-GRPO (v2)

The full SFT → SimPO → GRPO European post-training of Qwen3.5-9B — the "v2" recipe (full EU instruction set + reference-free preference optimisation + verifiable-reward RL). The final GRPO / RLVR stage uses European exam multiple-choice questions across 13 EU languages. Part of OpenEuroLLM Task 4.6, trained on LUMI (AMD MI250X / ROCm).

This repo ships the v2 weights — a clear upgrade over the earlier under-baked 9B.

Why RLVR: the reward is a deterministic verifier (did the model pick the correct option letter?) — no preference data, no reward model, and no translationese in the loop. It optimizes European-language correctness directly.

Results

Training reward (mcq_letter_exact, in-sample) climbed 0.51 → ~0.78 over 500 steps — the model learned to answer EU exam MCQs correctly ~78% of the time.

EU eval holdouts (oellm-eu-eval-holdouts-v1) — the v2 model scored on the large test_public split (5,472 rows), deterministic per-task scoring:

Overall: 64.9%, and — notably — even across all 38 European languages (56.9 – 71.5%), with no collapsed language. By task bucket:

The two zero buckets are known artifacts, not capability gaps: tool_calling — the 9B has no tool-use training (that capability lives in the 4B tool model); locale_formatting — a strict exact-match bucket that is chronically ~0. Excluding those two, the model averages ~83% on the remaining seven buckets.

(Full test_public split — this is the robust signal, not a small-bucket sample.)

Swedish capability (open-ended rubric)

A vLLM-free Swedish eval (Qwen3.5 isn't yet supported by EuroEval's fast backends): 24 prompts across 9 categories (sentiment, Swedish knowledge, reasoning, summarization, linguistic correctness, instruction-following, common sense, creative), scored 1–5 on språkkvalitet / korrekthet / instruktion / hjälpsamhet, plus automatic langdetect Swedish-purity.

The 9B is the strongest Swedish model in the lineup — best on every dimension. It gives the three largest lakes with areas, names Rayleigh-scattering for the blue sky, distinguishes universal vs particular premises in a logic question, and respects instruction constraints (4-item lists, an < 80-word formal email), all in fluent, fully-Swedish prose (purity 1.00). Tooling: scripts/eval_swedish_rubric.py + data/swedish_rubric_prompts.jsonl.

Training data

This model is the cumulative result of SFT → GRPO, so it carries the data from both stages (all openly-documented OpenEuroLLM Task 4.6 data; no proprietary data):

1. SFT stage (inherited from Qwen3.5-9B-EU-SFT) — ~400k examples (subset of ~1.08M), packed, bf16:

• General EU instructions — Dolci tulu3-euroblocks-85-15: EuroBlocks EU-multilingual instruction data (85%) + Tülu-3 (allenai/tulu-3) English replay (15%) — adds EU-language instruction-following while preserving English.

2. GRPO / RLVR stage (this model):

• birgermoell/oellm-eu-exam-mcq-v1 GRPO/RLVR split — European exam multiple-choice questions (EXAMS-QA, 13 EU languages, CC-BY-SA-4.0). The broader dataset spans ~35 languages, 28 sources (national, medical/ licensing, and academic exams), mixed licenses (filterable per row). Verifiable letter-match reward (mcq_letter_exact, β=0, no reward model) — optimizes EU-language answer correctness directly, with no translationese in the loop.

Evaluation — held out from training: birgermoell/oellm-eu-eval-holdouts-v1 — EU languages × task buckets, deterministic per-task scoring.

Training

Code & reproduction: https://github.com/BirgerMoell/qwen35-posttrain (docs/RUNBOOK.md, scripts/grpo_train.py, configs/grpo_qwen35_9b_exam.yaml).

Intended use & limitations

European-language assistant with improved exam-style/MCQ and reasoning accuracy. Same notes as the SFT model: it is a reasoning model (emits <think>…</think>; use enable_thinking=False for direct answers). No tool-use/locale training. RLVR optimizes the verifiable signal (answer correctness), not open-ended style.

License

Apache 2.0 (base: Qwen3.5-9B). Exam data: CC-BY-SA-4.0 (EXAMS-QA).

Citation

@misc{oellm-qwen35-9b-eu-sft-grpo,
  title  = {Qwen3.5-9B-EU-SFT-GRPO: RLVR on European exam MCQs},
  author = {Moëll, Birger and OpenEuroLLM Task 4.6},
  year   = {2026},
  url    = {https://github.com/BirgerMoell/qwen35-posttrain}
}