Downloads · 30 days
0
hadimh93/qwen2.5-gsm8k-grpo-dpo-adapters
qwen2.5-gsm8k-grpo-dpo-adapters is a machine learning model from hadimh93. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as mit.
Eight LoRA adapters (PEFT format) from the INLG 2026 paper Reasoning-Answer Entailment Under GRPO and DPO Fine-Tuning (Mohammadi and Giachanou, 2026): Qwen2.5-Instruct models at four scales, fine-tuned on GSM8K-derive…
Downloads · 30 days
0
Access
Public
Updated Aug 29, 2026
Repo size
1.9 GB
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 94%
From the Hugging Face model README
Eight LoRA adapters (PEFT format) from the INLG 2026 paper Reasoning-Answer Entailment Under GRPO and DPO Fine-Tuning (Mohammadi and Giachanou, 2026): Qwen2.5-Instruct models at four scales, fine-tuned on GSM8K-derived data with either GRPO or DPO under a single-GPU QLoRA budget.
Code, per-example evaluation data, and the evaluation pipeline: https://github.com/mohammadi-hadi/grpo_dpo_comparison
| Subfolder | Base model | LoRA r/α | Checkpoint |
|---|---|---|---|
grpo-1.5b | Qwen/Qwen2.5-1.5B-Instruct | 16/32 | 57 |
grpo-3b | Qwen/Qwen2.5-3B-Instruct | 16/32 | 114 |
grpo-7b | Qwen/Qwen2.5-7B-Instruct | 16/32 | 114 |
grpo-14b | Qwen/Qwen2.5-14B-Instruct | 16/32 | 114 |
dpo-1.5b | Qwen/Qwen2.5-1.5B-Instruct | 32/64 | 408 |
dpo-3b | Qwen/Qwen2.5-3B-Instruct | 32/64 | 408 |
dpo-7b | Qwen/Qwen2.5-7B-Instruct | 32/64 | 411 |
dpo-14b | Qwen/Qwen2.5-14B-Instruct | 32/64 | 274 |
from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "hadimh93/qwen2.5-gsm8k-grpo-dpo-adapters", subfolder="grpo-14b")
Adapters were trained with QLoRA on 4-bit quantized bases (unsloth/qwen2.5-*-instruct-unsloth-bnb-4bit). Applying them to full-precision bases is standard practice and is what the paper's bf16 re-evaluation does; minor numerical differences relative to 4-bit inference are possible.
GRPO expects reasoning between <start_working_out>/<end_working_out> tags and the answer between <SOLUTION> tags; DPO uses <reasoning>/<answer> tags. See the GitHub repository for the exact prompt templates.