qwen-reasoning-grpo-4bit

Qwen 2.5 1.5B - GRPO Reasoning Model (4-bit)

This is a 4-bit quantized version of Qwen2.5-1.5B fine-tuned with GRPO (Generative Reward Policy Optimization) for enhanced mathematical and logical reasoning.

Model Description

• Base Model: Qwen/Qwen2.5-1.5B-Instruct
• Training Method: GRPO (Reinforcement Learning)
• Training Dataset: GSM8K (Math word problems)
• Quantization: 4-bit NF4 (bitsandbytes)
• Training Steps: 40
• Format: Uses <reasoning> tags for chain-of-thought

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mhmdelbadry1/qwen-reasoning-grpo-4bit"

# Load model (4-bit quantized)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
    load_in_4bit=True
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

# Inference
prompt = "Solve the equation: 2x + 5 = 15. Show your steps."
messages = [{
    "role": "user",
    "content": prompt + "\nPlease reason step-by-step. You must use <reasoning> tags (not <think>) for your thought process and <answer> tags for the final result."
}]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    temperature=0.7,
    top_p=0.9
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Training Details

GRPO Reward Functions

1. XML Format Reward (1.0): Correct use of <reasoning> and <answer> tags
2. Correctness Reward (2.0): Answer matches ground truth
3. Soft Format Reward (0.2): Partial credit for attempting structure

Performance

• Format Compliance: 100% (uses XML tags consistently)
• Reasoning Depth: Detailed step-by-step explanations

Limitations

• Optimized for math/logic tasks (GSM8K training)
• May over-apply mathematical reasoning to linguistic puzzles
• 4-bit quantization may slightly reduce accuracy vs full precision

Citation

If you use this model, please cite:

@misc{qwen-reasoning-grpo-2024,
  title={Qwen 2.5 GRPO Reasoning Model},
  author={Mohamed Elbadry},
  year={2025},
  howpublished={\url{https://huggingface.co/mhmdelbadry1/qwen-reasoning-grpo-4bit}}
}

License

Apache 2.0 (inherited from base model)