Downloads · 30 days
8
11% of all-time downloads
yavuz-ai/qwen2.5-3b-dpo-finqa
qwen2.5-3b-dpo-finqa is a text generation model from yavuz-ai. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
A LoRA policy from DPO on FinQA, where the preference pairs come from the same verifiable reward as the GRPO arm: sample completions from the base, label each correct/incorrect by whether the final number is right, pa…
Downloads · 30 days
8
11% of all-time downloads
All-time downloads
76
Public
Repo size
131 MB
Likes
0
Public
Click a slice to open those files.
.safetensors120 MB · 91%
From the Hugging Face model README
A LoRA policy from DPO on FinQA, where the preference pairs come from the same verifiable reward as the GRPO arm: sample completions from the base, label each correct/incorrect by whether the final number is right, pair a correct one (chosen) vs an incorrect one (rejected). This is the offline counterpart in finqa-rlvr - same base, same signal as GRPO, only the optimization differs.
FinQA validation accuracy (greedy, n=200): 0.530 -> 0.555 (+5 questions). A small positive trend, but within noise at this eval size (per-arm SE ~3.5 pts). The project's honest conclusion is not "DPO beats GRPO" - it is that the method is second-order at this scale; reward design and base headroom dominate. Trained on yavuz-ai/finqa-prefs (1,881 correctness-labelled pairs).
DPO (TRL, reference-free with the LoRA adapter), LoRA r=16 / alpha=32, beta 0.1, lr 5e-6, 1 epoch, 1x A100.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen2.5-3b-dpo-finqa")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")