mistral-7b-arc-dpo

Mistral-7B ARC-Challenge SFT + Synthetic-Pair DPO

Mistral-7B-v0.1 fine-tuned for the ARC-Challenge benchmark (Upstage AIRE deep-learning coding test). acc_norm 71.84 ± 0.48 under the official protocol (lm-evaluation-harness arc_challenge, 25-shot, four few-shot seeds), vs. 61.07 ± 0.36 for the base model measured identically. The task's reference score to beat was 61.43.

Recipe

Two stages, both LoRA (r=32, alpha=64, all seven projections), merged into bf16 for this checkpoint:

1. SFT on ARC-Challenge + ARC-Easy train splits (3,370 items), rendered byte-for-byte in the harness's own surface form (Question: ...\nAnswer:), loss masked to the answer span. lr 1e-4 cosine, 2 epochs.
2. DPO on 8,025 gold-vs-distractor pairs drawn from a synthetic pool: 2,675 teacher-generated (Qwen3.8-27B), label-verified, ARC-form-matched MCQs that the SFT model still failed ("hard mining"). beta 0.1, with hard-pair gating and a chosen-NLL anchor.

The synthetic pool and its generation/verification pipeline are published separately (see the companion dataset repo), including the contamination audit: generated items were screened against ARC-C/E test+validation by exact and 8-gram overlap; the teacher was only ever shown ARC train items.

Evaluation

lm_eval --model hf --model_args pretrained=<this repo>,dtype=float16 \
  --tasks arc_challenge --num_fewshot 25 --seed 0,1234,1234,<fewshot_seed>

Trained on 2x RTX A6000; the SFT stage also reproduces on a single T4 in its QLoRA variant (67.66 ± 0.36, statistically equivalent to bf16 SFT).