Downloads · 30 days
4
1% of all-time downloads
shubhamrgandhi/Qwen3-4B-DPO-prm-pairs
Qwen3-4B-DPO-prm-pairs is a text generation model from shubhamrgandhi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
DPO fine-tune of Qwen/Qwen3-4B-Instruct-2507 on preference pairs of supervisor feedback for an LLM coding agent. The task the model is trained toward: given an agent's trajectory, produce trajectory-level error analys…
Downloads · 30 days
4
1% of all-time downloads
All-time downloads
486
Public
Parameters
4B
8.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8 GB · 100%
From the Hugging Face model README
DPO fine-tune of Qwen/Qwen3-4B-Instruct-2507 on preference pairs of supervisor feedback for an LLM coding agent. The task the model is trained toward: given an agent's trajectory, produce trajectory-level error analysis and corrective guidance, preferring higher-quality critiques.
Trained with TRL's DPOTrainer.
The run improved over its own baseline but did not exceed chance on pairwise preference accuracy. Reported honestly:
| epoch | eval_loss | eval_rewards/accuracies | eval_rewards/margins |
|---|---|---|---|
| 0.00 (baseline) | 0.6914 | 0.0000 * | 0.0000 |
| 0.23 | 0.7172 | 0.3187 | −0.0073 |
| 0.46 | 0.6926 | 0.3875 | +0.0746 |
| 0.69 | 0.6925 | 0.4562 | +0.0866 |
| 0.92 | 0.6949 | 0.4313 | +0.0877 |
| 1.00 (final) | 0.6680 | 0.4375 | +0.1310 |
* At step 0 the policy is the reference model, so both rewards are exactly 0
and the strict chosen > rejected comparison is False for every pair. That 0 is
a degenerate tie, not a meaningful score.
Final train_loss 0.666.
Interpretation. Eval loss ended below baseline (0.668 vs 0.6914) and the
reward margin grew steadily to +0.131, so the model does separate some pairs.
But rewards/accuracies finished at 0.4375, below the 0.5 chance line — a
positive mean margin combined with a sub-chance win rate means the average is
carried by a minority of strongly-separated pairs while most pairs remain
slightly mis-ranked.
The metric curve was still climbing at the end of epoch 1, so this looks
under-trained rather than converged: only 44 optimizer steps were taken at
learning_rate=5e-7. Treat this as a checkpoint of record, not a model that
reliably beats its base at preference ranking.
trl.DPOTrainer), β = 0.1 (default)messages + chosen/rejected),
split 90/10 → 1,386 train / 155 eval (seed 42). Pairs are pre-filtered to fit
whole within 8,192 tokens; median length ≈ 7.7k tokens.per_device_train_batch_size=1 × gradient_accumulation_steps=4
× 8 GPUs = effective batch 32 → 44 optimizer steps for 1 epochPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \
--config_file examples/accelerate_configs/deepspeed_zero3.yaml \
dpo_prm_pairs.py
gradient_accumulation_steps to restore ~87 steps) or a higher learning rate
is the obvious next experiment.