Downloads · 30 days
4
4% of all-time downloads
Abdullah121212/qwen3-0.6b-dpo-best
qwen3-0.6b-dpo-best is a text generation model from Abdullah121212. Use it when you need the model to write or continue text. It is set up for peft.
Best DPO (Direct Preference Optimization) LoRA adapter for Qwen/Qwen3-0.6B-Base, built on top of the best SFT adapter. Trained on the UltraFeedback Binarized dataset.
Downloads · 30 days
4
4% of all-time downloads
All-time downloads
95
Public
Repo size
29.8 MB
Likes
0
Public
Click a slice to open those files.
.safetensors18.4 MB · 62%
From the Hugging Face model README
Best DPO (Direct Preference Optimization) LoRA adapter for Qwen/Qwen3-0.6B-Base, built on top of the best SFT adapter. Trained on the UltraFeedback Binarized dataset.
| Parameter | Value |
|---|---|
| DPO beta | 0.05 |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| Learning rate | 5e-5 |
| Batch size | 2 |
| Gradient accumulation | 2 |
| Epochs | 1 |
| Dropout | 0.05 |
Evaluated on 10 manually curated prompts with gold-reference answers.
| Stage | BLEU | BERTScore F1 |
|---|---|---|
| Base (Qwen3-0.6B) | 9.82 | 0.8803 |
| SFT (best) | 9.47 | 0.8836 |
| DPO (best — this model) | 11.94 | 0.8863 |
pip install --upgrade torchao transformers peft accelerate
Note: You may encounter
ImportError: Found an incompatible version of torchaoif yourtorchaoversion is below 0.16.0. Runpip install --upgrade torchaoto fix this.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B-Base", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Abdullah121212/qwen3-0.6b-dpo-best")
model = PeftModel.from_pretrained(base_model, "Abdullah121212/qwen3-0.6b-dpo-best")
model.eval()
prompt = "Explain the difference between supervised and unsupervised learning in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Baseline (Qwen3-0.6B-Base)
-> SFT (best trial 3 — see Abdullah121212/qwen3-0.6b-sft-best)
-> DPO (5 trials, this is trial 3 = best)