SmolGRPO-135M

SmolGRPO-135M

A fine-tuned version of HuggingFaceTB/SmolLM-135M-Instruct trained with Group Relative Policy Optimization (GRPO) to produce concise summaries (~50 tokens).

Training details

• Algorithm: GRPO via trl 0.14.0
• Base model: HuggingFaceTB/SmolLM-135M-Instruct
• Fine-tuning method: LoRA (r=16, alpha=32, target: q/k/v/o_proj)
• Dataset: mlabonne/smoltldr
• Reward: Negative absolute deviation from 50-token target length
• Num generations per prompt: 4
• Training epochs: 1

Reward function

def reward_token_length(completions, target_length=50, **kwargs):
    return [-abs(target_length - len(tokenizer.encode(c, add_special_tokens=False)))
            for c in completions]

Limitations

• Trained for length control, not factual accuracy. May produce fluent but incorrect summaries.
• Target length is 50 tokens. Prompts requiring much longer responses are handled less well.
• Training data is English-only.