qwen-1.5b-grpo-math

Qwen2.5-1.5B GRPO Math

A Qwen2.5-1.5B-Instruct model fine-tuned with GRPO (Group Relative Policy Optimization) on the GSM8K math dataset. The model learned to solve grade school math word problems through reinforcement learning — no human-labeled reasoning chains, just a binary correctness signal.

What is GRPO?

GRPO is the same RL technique used to train DeepSeek-R1. Instead of learning from labeled step-by-step solutions, the model:

1. Generates multiple answers (group of 4) for each math problem
2. Checks correctness — does the final number match the ground truth?
3. Learns from relative performance — within each group, correct answers get higher advantage, wrong ones get lower

This teaches the model to reason and self-verify without explicit reasoning supervision.

Training Details

Performance

The model went from solving 1 in 4 math problems to 4 in 5 through pure RL training.

Training Curves

• Training Loss: Converges near zero after initial policy updates
• Mean Reward: Jumps from 0.25 to ~0.80 within the first 200 steps, stabilizes at 0.80-0.90
• Learning Rate: Cosine decay from 5e-6 to 0
• Reward Std: Stays non-zero (~0.25) throughout — healthy gradient signal, no reward collapse

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model + LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base_model, "usama10/qwen-1.5b-grpo-math")
tokenizer = AutoTokenizer.from_pretrained("usama10/qwen-1.5b-grpo-math")

messages = [
    {"role": "system", "content": "You are a math tutor. Solve the problem step by step, then give the final numerical answer after ####."},
    {"role": "user", "content": "A store sells apples for $2 each and oranges for $3 each. If Sarah buys 5 apples and 3 oranges, how much does she spend?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Reward Function

The reward function extracts the final numerical answer after #### from the model's output and compares it to the ground truth:

def correctness_reward(completion, ground_truth):
    predicted = extract_number_after_hashes(completion)
    expected = extract_number_after_hashes(ground_truth)
    return 1.0 if predicted == expected else 0.0

Limitations

• Trained on grade school math only (GSM8K) — may not generalize to advanced math
• Single epoch of training — more epochs or larger models could improve performance
• LoRA adapter — needs base model to run inference