Qwen2.5-3B-GRPO-NuminaMath-100K

Qwen2.5-3B-GRPO-NuminaMath-100K

This model is a fine-tuned version of ermiaazarkhalili/Qwen2.5-3B-SFT-UltraChat trained on the AI-MO/NuminaMath-CoT dataset using GRPO with LoRA adapters.

Overview

Qwen2.5-3B-GRPO-NuminaMath-100K is a language model optimized using GRPO. Group Relative Policy Optimization (GRPO) is an online RL method that optimizes policy using group-relative rewards without a critic model.

Key Features

• Reinforcement Learning: Trained with Group Relative Policy Optimization
• Mathematical Reasoning: Optimized on N/A math problems from NuminaMath-CoT
• Reward Function: Uses combined reward (accuracy + format)
• Efficient Training: Uses LoRA with 4-bit quantization
• Optimized for Inference: Available in GGUF quantizations for local deployment

Model Details

Training Information

Training Configuration

LoRA Configuration

Training Metrics

Dataset

This model was trained on the AI-MO/NuminaMath-CoT dataset.

Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the sum of 2 + 2?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(response)

Using Pipeline

from transformers import pipeline

generator = pipeline("text-generation", model="ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K", device_map="auto")
messages = [{"role": "user", "content": "Explain the concept of machine learning."}]
output = generator(messages, max_new_tokens=256, return_full_text=False)
print(output[0]["generated_text"])

4-bit Quantized Inference

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    "ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K",
    quantization_config=quantization_config,
    device_map="auto"
)

GGUF Versions

For CPU or mixed CPU/GPU inference, GGUF quantized versions are available at: ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF

Using with Ollama

ollama pull hf.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF:Q4_K_M
ollama run hf.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF:Q4_K_M "Hello!"

Limitations

• Language: Primarily trained on English data
• Knowledge Cutoff: Limited to base model's training data cutoff
• Hallucinations: May generate plausible-sounding but incorrect information
• Context Length: Fine-tuned with 2,048 token limit
• Safety: Not extensively safety-tuned; use with appropriate guardrails

Intended Use

Recommended Uses

• Research on language model fine-tuning
• Educational purposes
• Personal projects
• Prototyping conversational AI

Out-of-Scope Uses

• Production systems without additional safety measures
• Medical, legal, or financial advice
• Generating harmful or misleading content

Training Framework

• TRL: 0.24.0
• Transformers: 4.57.3
• PyTorch: 2.9.0
• Datasets: 4.3.0
• PEFT: 0.18.0
• BitsAndBytes: 0.49.0

Citation

@misc{ermiaazarkhalili_qwen2.5_3b_grpo_numinamath_100k,
    author = {ermiaazarkhalili},
    title = {Qwen2.5-3B-GRPO-NuminaMath-100K: Fine-tuned Qwen2.5-3B-SFT-UltraChat on NuminaMath-CoT},
    year = {2026},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K}}
}

Acknowledgments

• Base model developers at ermiaazarkhalili
• Hugging Face TRL Team for the training library
• Dataset creators and contributors
• Compute Canada / DRAC for HPC resources

Contact

For questions or issues, please open an issue on the model repository.