LFM2-700M-GRPO-NuminaMath-10K

LFM2-700M-GRPO-NuminaMath-10K

This model is a fine-tuned version of LiquidAI/LFM2-700M trained on the AI-MO/NuminaMath-CoT dataset using Group Relative Policy Optimization (GRPO) - an online reinforcement learning method.

Overview

LFM2-700M-GRPO-NuminaMath-10K is optimized for mathematical reasoning tasks. It uses GRPO to learn from reward signals based on answer correctness and format adherence, enabling it to generate more accurate step-by-step solutions.

Key Features

• Reinforcement Learning: Trained with GRPO for improved reasoning capabilities
• Math Focus: Optimized on 10,000 math problems from NuminaMath-CoT
• Multi-Sample Learning: Uses 2 generations per prompt for robust training
• Combined Reward: Evaluates both answer accuracy and output format

Model Details

Training Information

GRPO Configuration

LoRA Configuration

Training Metrics

Reward Function

The model uses a combined reward function:

1. Math Accuracy: Extracts and validates final numerical answers
2. Format Compliance: Checks for proper step-by-step reasoning format
3. Combined Score: Weighted combination of accuracy and format rewards

Training Hardware

• GPU: NVIDIA H100 40GB MIG (3g.40gb)
• CPU: 8 vCPUs
• Memory: 64GB
• Platform: Compute Canada (Fir Cluster)

Dataset

This model was trained on the AI-MO/NuminaMath-CoT dataset:

NuminaMath-CoT provides step-by-step mathematical solutions, enabling the model to learn structured reasoning patterns.

Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ermiaazarkhalili/LFM2-700M-GRPO-NuminaMath-10K"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Math problem
prompt = "Solve step by step: If a train travels 120 km in 2 hours, what is its average speed?"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations

• Domain Specific: Optimized for math; may not generalize to other reasoning tasks
• Language: English only
• Hallucinations: May produce incorrect calculations despite correct format
• Verification Needed: Always verify mathematical results independently

Intended Use

Recommended Uses

• Mathematical problem solving
• Step-by-step reasoning demonstrations
• Educational math tutoring applications
• Research on RL-trained language models

Out-of-Scope Uses

• Critical calculations requiring absolute accuracy
• Non-mathematical reasoning tasks
• Production systems without verification

Citation

@misc{ermiaazarkhalili_lfm2_700m_grpo_numinamath_10k,
    author = {Ermia Azarkhalili},
    title = {LFM2-700M-GRPO-NuminaMath-10K: GRPO-trained LFM2-700M for Math},
    year = {2025},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/ermiaazarkhalili/LFM2-700M-GRPO-NuminaMath-10K}}
}

Acknowledgments

• LiquidAI for the LFM2 base model
• Hugging Face TRL Team for the GRPO implementation
• NuminaMath dataset creators
• Compute Canada for providing HPC resources