Downloads · 30 days
8
100% of all-time downloads
DPO-4B-LiteOS-fork596
HaoranLiu/DPO-4B-LiteOS-fork596
DPO-4B-LiteOS-fork596 is a machine learning model from HaoranLiu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Trajectory-level DPO on fork-point (prefix-aligned) pairs, from HaoranLiu/SFT-4B-LiteOS. Full forkturn 0 set, 1 epoch, 149 steps.
All-time downloads
8
Public
Parameters
4.4B
8.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.9 GB · 100%
At a glance
- License
- other
- Model type
- qwen3_vl
- Access
- Public
- Created
- Aug 16, 2026
- Updated
- Aug 16, 2026
- SHA
- 398f33b3
Base models
Model card
From the Hugging Face model README
DPO-4B-LiteOS-fork596
Trajectory-level DPO on fork-point (prefix-aligned) pairs, from HaoranLiu/SFT-4B-LiteOS.
Full fork_turn > 0 set, 1 epoch, 149 steps.
- data:
HaoranLiu/DPO-fork-Qwen, all 596 pairs withfork_turn > 0 - chosen:
gpt-5.5fork continuation · rejected:Qwen/Qwen3-VL-4B-Instruct
Hyperparameters
| objective | trajectory-level DPO, summed per-action log-probs (no length normalization) |
| dpo_beta | 0.1 |
| kl_coef | 0.0 |
| optimizer | Adam, beta1 0.9, beta2 0.95, eps 1e-8 |
| lr | 5e-7, cosine decay to 0, warmup fraction 0.1 |
| weight_decay | 0.1 |
| clip_grad | 1.0 |
| global_batch_size | 4 pairs/step |
| micro_batch_size | 1 |
| seq_length | 4096 |
| precision | bf16 |
| seed | 1234 |
| parallelism | TP=2, PP=1, CP=1, DP=1 (2xH100), sequence_parallel |
| optimizer sharding | distributed (ZeRO-1) + CPU offload, precision-aware |
| recompute | full, uniform, num_layers 1 |
| loss / advantage hooks | lite.train.dpo.dpo_loss_function / lite.train.dpo.compute_dpo_pair_weights |
Results
lite.osworld eval, 332 tasks, greedy, max_steps 30, group_size 1. Mean over all 332.
| model | steps | mean | success | success_rate |
|---|---|---|---|---|
| SFT-4B-LiteOS (init) | — | 0.3220 | 103 | 31.02% |
| this ckpt — fork596 (1 ep) | 149 | 0.3557 | 116 | 34.94% |
| fork350 ep1 | 88 | 0.3628 | 116 | 34.94% |
| fork350 ep2 | 176 | 0.3571 | 115 | 34.64% |
| fork350 ep3 | 261 | 0.3406 | 110 | 33.13% |
| naive pairs ep1 | 96 | 0.3579 | 117 | 35.24% |
| naive pairs ep2 | 192 | 0.3801 | 123 | 37.05% |
| naive pairs ep3 | 285 | 0.3571 | 116 | 34.94% |
| 2B-negatives ep1 | 102 | 0.3581 | 117 | 35.24% |
| 32B-negatives ep1 | 84 | 0.3466 | 111 | 33.43% |
| mixed 2B+4B+32B ep2/3 | 188 | 0.3746 | 122 | 36.75% |
| first-diverge (loss on fork turn only) ep2 | 388 | 0.2718 | 87 | 26.20% |
- Type
- qwen3_vl
- License
- other
- Created
- Aug 16, 2026
- Updated
- Aug 16, 2026