qwen2.5-3b-yoda-checkpoints

Qwen2.5-3B Yoda persona — SFT + RLAIF (DPO) checkpoints

Merged full-model checkpoints from CS 2881R HW1 (team chen-qi-tewari): teaching Qwen2.5-3B-Instruct a Yoda persona via SFT, then improving it with RLAIF — judge-guided DPO (AI judge: Qwen3-8B scoring K=4 sampled candidates per prompt; best/worst pairs, LoRA DPO with the SFT model as reference).

Each subfolder is a complete merged model, loadable with:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "samc1249/qwen2.5-3b-yoda-checkpoints"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="dpo_v1_beta001")
tok = AutoTokenizer.from_pretrained(repo, subfolder="dpo_v1_beta001")

Headline models

Reference points (not in this repo): base Qwen2.5-3B-Instruct scores 2.96 judge / 85.0 GSM8K / 63.4 MATH-500; base with a Yoda system prompt scores 3.66 / 85.0 / 64.4. Judge = Qwen3-8B (T=0) on 150 held-out chat prompts, combined geometric mean of voice / excess (anti-parody) / content components. STEM = greedy decoding, rule-based verifier, frozen 1000-problem sets.

License

Derivative of Qwen/Qwen2.5-3B-Instruct, distributed under the Qwen Research License. Research use only.

Revised CP1 models (sft_v2, three seeds)

sft_v2 is a rebuilt CP1: the 800 terse answer-key maths training rows were replaced with full-length Yoda-voiced solutions (the base model's own reasoning, restyled); the 450 chat rows and all hyperparameters are unchanged. Trained at three seeds. Seed 0 is the one used downstream — the default, not the best scorer.

For comparison, sft_v1 scores 62.2 GSM8K / 36.8 MATH-500 and speaks 0% Yoda in its maths answers (0.00 inversions/sentence); base scores 85.0 / 63.4. Accuracies are lower bounds: ~7.5% of answers hit the 1,024-token generation cap. Greedy decoding, rule-based verifier, frozen held-out sets.

Revised CP2 model (cp2_dpo)

cp2_dpo is preference training (DPO, β=0.1, ~400 pairs, 2 epochs) on top of sft_v2_s0, with pairs picked by a pairwise AI judge (Qwen3-8B): two sampled answers per training question, judged twice with order swapped, wrong maths answers vetoed by the verifier before the judge sees them.

The win rate is the judge's preference on 150 held-out chat questions, judged in both orders (72 wins, 11 losses, 67 unclear; 50% = no change; base scores 20.3% on the same measure). Maths at the 2,048-token cap: 72.3% overall vs sft_v2_s0's 71.6% in the same run (paired change +0.7, 95% range −1.6 to +3.0). Known caveats, reported with the result: the same judge built the training pairs and scored the win rate; judge-vs-human agreement on a 40-pair blind check was at chance (0.45); chat answers grew ~25% longer.

Revised CP3 model (cp3_grpo)

cp3_grpo is GRPO (TRL, 100 steps, LR 3e-5, KL leash 0.04, 8 answers per question at temperature 0.9) on top of cp2_dpo, with a multi-objective reward: a rule-based maths verifier (wrong answers score 0 and never reach the judge), plus half a point when the pairwise judge (Qwen3-8B, both orders) prefers the answer's Yoda voice over CP1's reference answer.

Paired against cp2_dpo in the same run: maths +0.3 (95% range −1.9 to +2.5; 65 questions right only after, 62 only before) — capability held. Chat voice change vs CP2's 70.3% is −5.0 (95% −11.0 to +1.0) — no clear persona effect from GRPO on top of DPO; maths voice edged up (56.6% → 58.0%). Training diagnostics: verifier score 0.758 → 0.778, KL to the start model 0.0027, 0.7% of answers cut off — no reward-hacking signals. The run crashed once at step 82 on a transient NaN and resumed cleanly from checkpoint-80 (D-99). Same caveats as CP2: the scoring judge also shaped the reward, and its agreement with a 40-pair human check is at chance (0.45).