Downloads · 30 days
14
14% of all-time downloads
Keitsuna123/llama-3.2-1b-fc-grpo
llama-3.2-1b-fc-grpo is a reinforcement learning model from Keitsuna123. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as llama3.2.
Code & writeup: https://github.com/keitake123/llama-function-calling-study
Downloads · 30 days
14
14% of all-time downloads
All-time downloads
100
Public
Parameters
1.2B
2.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.5 GB · 99%
From the Hugging Face model README
Code & writeup: https://github.com/keitake123/llama-function-calling-study
GRPO (RL post-training) applied to the v1 SFT model, aiming to recover the irrelevance-detection regression that SFT introduced. This is a negative result — reported honestly as part of a four-model study.
| Category | v1 SFT | This model (GRPO) | Path B (refusal-SFT) |
|---|---|---|---|
| irrelevance | 5.8 | 5.4 | 21.7 |
| live_irrelevance | 16.9 | 17.2 | 36.8 |
| simple_python | 77.5 | 77.2 | 78.2 |
| multiple | 74.0 | 74.0 | 72.0 |
| live_simple | 57.0 | 56.2 | 54.3 |
| live_multiple | 38.8 | 39.4 | 35.7 |
| live_relevance | 93.8 | 93.8 | 81.2 |
Finding: GRPO with a verifiable, irrelevance-penalizing reward produced no measurable irrelevance recovery (5.8→5.4). Hypothesis: GRPO can only amplify behaviors the model already samples. The SFT "always-call" prior was so strong that the model essentially never sampled a refusal during rollouts, so all completions scored similarly and there was no gradient signal to reinforce refusal. Supervised demonstration (Path B) succeeded where RL did not.
An earlier GRPO attempt also revealed a reward-hacking failure mode: the model collapsed to degenerate repetitive output that exploited a gap in the reward function (unparseable output was scored as a valid refusal on irrelevance cases). This was fixed by hardening the reward with adversarial test cases before the final run.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("Keitsuna123/llama-3.2-1b-fc-grpo")
model = AutoModelForCausalLM.from_pretrained(
"Keitsuna123/llama-3.2-1b-fc-grpo", torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get the weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}
}}]
text = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
| Model | Description |
|---|---|
| fc-sft-full | v1 SFT on xLAM |
| fc-sft-v2-merged | SFT on xLAM + distilabel (data-scaling ablation) |
| fc-grpo | GRPO for irrelevance recovery (this model) |
| fc-pathb | Supervised refusal training (effective irrelevance fix) |
@misc{taketsuna2026_fc_smallmodel,
title = {Small-Model Function Calling: Comparing SFT, Data Scaling, GRPO, and Supervised Refusal Training},
author = {Taketsuna, Keiichi},
year = {2026},
howpublished = {\url{https://huggingface.co/Keitsuna123}},
note = {Llama-3.2-1B function-calling post-training study on BFCL v4}
}