Llama-3.1-8B-RL-math-400-step

Uploaded model

• Developed by: wanderly0501
• License: apache-2.0
• Finetuned from model : unsloth/meta-llama-3.1-8b-instruct-unsloth-bnb-4bit
• Dataset: openai/gsm8k

Training details and output

Prompt/ouput format

The model is prompted (system prompt) to respond in a fixed XML-like structure:

<reasoning>
...
</reasoning>
<answer>
...
</answer>

Model & LoRA setup

• Loaded in 4-bit with FastLanguageModel (Unsloth), fast_inference=True (vLLM backend) for GRPO rollout generation
• LoRA rank 32, lora_alpha=32, targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• max_seq_length=1972, max_prompt_length=986, max_completion_length=986

GRPO training config

• learning_rate=5e-6, cosine schedule, 10% warmup
• optim=paged_adamw_8bit, max_grad_norm=0.1
• per_device_train_batch_size=8, gradient_accumulation_steps=6
• num_generations=8 (group size — completions sampled per prompt)
• max_steps=400

Results

Evaluated greedy (temperature=0.0) on held-out test sets, grading on exact match of the extracted <answer> content:

This llama model was trained 2x faster with Unsloth

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>