Qwen2.5-3B-GRPO-16bit

Qwen2.5-3B-GRPO-16bit (LoRA merged)

Model Summary

This model is a fine-tuned version of Qwen2.5-3B-Instruct using GRPO (Group Relative Policy Optimization) with multiple reward functions to encourage structured reasoning and correct answers on math word problems (GSM8K). It has been trained in 16-bit merged LoRA format for easier inference and deployment.

• Base model: Qwen/Qwen2.5-3B-Instruct
• Fine-tuning method: GRPO with multiple reward functions
• Format enforced: XML-style <reasoning> and <answer> blocks
• Optimized for: step-by-step reasoning and correctness on grade-school math problems
• Precision: 16-bit (merged LoRA)
• Frameworks: Unsloth, vLLM, Hugging Face Transformers

---

Training Details

Objective

The model was trained to produce structured answers in the following format:

<reasoning>
Step-by-step reasoning here...
</reasoning>
<answer>
Final concise answer here
</answer>

Dataset

• GSM8K (main split) for grade-school math word problems.
• Dataset preprocessing enforced extraction of gold answers and applied a chat-style template with a system prompt.

Reward Functions

The GRPO training used multiple rewards:

• XML format adherence (strict & soft checks)
• Correctness reward (matching gold answers)
• Integer-answer reward (to encourage numeric outputs)
• Count-based XML structure score (penalizes malformed completions)

Hyperparameters

• Sequence length: 1024
• LoRA rank: 64
• Batch size: 1 per device (with gradient accumulation = 1)
• Learning rate: 5e-6 (cosine schedule, warmup 0.1)
• Optimization: AdamW 8-bit, weight decay 0.1
• Max steps: 250 (early experiments, short run)
• Generation config: temperature 0.8, top-p 0.95, max tokens 1024

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("srikar-v05/Qwen2.5-3B-GRPO-16bit")
tokenizer = AutoTokenizer.from_pretrained("srikar-v05/Qwen2.5-3B-GRPO-16bit")

messages = [
    {"role": "system", "content": "Respond in <reasoning>...</reasoning><answer>...</answer> format."},
    {"role": "user", "content": "Sarah has 12 apples. She gives 4 apples to her friend and eats 2 herself. How many apples does Sarah have left?"},
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0]))

---

Example Output

Input:

How many r’s are in strawberry?

Output:

<reasoning>
The word "strawberry" has 10 letters. The letter "r" appears 3 times.
</reasoning>
<answer>
3
</answer>

---

Limitations & Bias

• The model is optimized for math word problems and may not generalize perfectly to other domains.
• It may produce hallucinations if prompted outside of math or reasoning tasks.
• Training was limited to ~250 steps, so performance gains are modest. Longer training could yield stronger results.

---

Intended Use

• Research on reinforcement learning with rewards for structured reasoning.
• Educational tools for grade school math tutoring.
• Demonstration of GRPO + LoRA + vLLM workflows.

Not intended for: high-stakes decision-making, sensitive applications, or domains requiring factual accuracy beyond math reasoning.

---