Downloads · 30 days
30
4% of all-time downloads
Keitsuna123/llama-3.2-1b-fc-dpo
llama-3.2-1b-fc-dpo is a text generation model from Keitsuna123. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
The v1 SFT model further trained with DPO (Direct Preference Optimization) to recover an irrelevance-detection regression. DPO is a preference optimization method — not reinforcement learning — that optimizes a model…
Downloads · 30 days
30
4% of all-time downloads
All-time downloads
838
Public
Parameters
1.2B
2.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.5 GB · 99%
From the Hugging Face model README
The v1 SFT model further trained with DPO (Direct Preference Optimization) to recover an irrelevance-detection regression. DPO is a preference optimization method — not reinforcement learning — that optimizes a model to prefer "chosen" over "rejected" responses directly, without a reward model or sampling. This is one of six models in a study comparing supervised, preference, and RL post-training for small-model function calling.
Code & writeup: https://github.com/keitake123/llama-function-calling-study
| Category | Base | v1 SFT | GRPO (RL) | DPO (pref.) | Path B (SFT) |
|---|---|---|---|---|---|
| irrelevance | 35.8 | 5.8 | 5.4 | 8.3 | 21.7 |
| live_irrelevance | 67.3 | 16.9 | 17.2 | 28.6 | 36.8 |
| simple_python | 75.0 | 77.5 | 77.2 | 77.5 | 78.2 |
| multiple | 50.5 | 74.0 | 74.0 | 73.5 | 72.0 |
| live_simple | 31.8 | 57.0 | 56.2 | 58.5 | 54.3 |
| live_multiple | 7.3 | 38.8 | 39.4 | 37.0 | 35.7 |
| live_relevance | 43.8 | 93.8 | 93.8 | 81.2 | 81.2 |
Finding: DPO produced partial irrelevance recovery (5.8→8.3 non-live; 16.9→28.6 live) — clearly better than GRPO (which showed no recovery) but below refusal-SFT. This fits a clean pattern across the three methods:
Recovery scaled with how directly each method exposed the model to the target behavior. GRPO never samples a refusal (the SFT always-call prior is too strong) → no signal → no recovery. DPO is shown refusals as the "chosen" response → partial recovery, at low cost to call categories. Refusal-SFT trains directly on refusals → strongest recovery.
Ordering: imitation (SFT) > preference (DPO) > RL (GRPO). RL recovers least because it cannot reinforce a behavior absent from the policy's sampling distribution.
Notably, DPO preserved call-required categories well (simple_python, live_simple held up), giving arguably the best precision/recall balance despite a lower raw irrelevance score than Path B.
DPOTrainer), LoRA (r=16, α=32)Note on stability: an initial run with beta=0.1 and a stacked (unmerged) reference adapter caused output-format drift (the model emitted a malformed call schema and occasionally code). Fixed by (1) cleanly merging the v1 adapter into the base before DPO so the reference model is well-formed, and (2) raising beta to 0.3 for a stronger anchor. See the repo's LESSONS.md.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("Keitsuna123/llama-3.2-1b-fc-dpo")
model = AutoModelForCausalLM.from_pretrained(
"Keitsuna123/llama-3.2-1b-fc-dpo", torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get the weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}
}}]
text = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
| Model | Method | Description |
|---|---|---|
| fc-sft-full | SFT | v1 SFT on xLAM |
| fc-sft-v2-merged | SFT | + distilabel (data-scaling ablation) |
| fc-grpo | RL | GRPO (no recovery) |
| fc-pathb | SFT | refusal-SFT (best recovery) |
| fc-dpo | DPO | preference optimization (this model) |
@misc{taketsuna2026_fc_smallmodel,
title = {Small-Model Function Calling: Comparing SFT, Data Scaling, GRPO, and DPO Post-Training},
author = {Taketsuna, Keiichi},
year = {2026},
howpublished = {\url{https://github.com/keitake123/llama-function-calling-study}},
note = {Llama-3.2-1B function-calling post-training study on BFCL v4}
}