GRPO-TCR-Qwen3-4B-test

GRPO-TCR-Qwen3-4B-test

This repository contains a full fine-tuned model based on Qwen3-4B-Instruct-2507, trained in two stages:

1. SFT (Supervised Fine-Tuning) — multi-turn agentic cold-start
2. GRPO-TCR (Group Relative Policy Optimization with Tool Call Reward) — reinforcement learning

Training uses the Open-AgentRL framework based on the DemyAgent methodology.

Note: This is a test run with minimal data (8 train / 4 test samples, 6 steps) to validate the training pipeline configuration.

Training Objective

This model is trained to perform deliberative agentic reasoning — selectively calling a code_interpreter tool across multiple turns to solve math and coding problems, rather than relying on verbose self-reasoning or indiscriminate tool calls.

The GRPO-TCR stage reinforces:

• Correct final answers (outcome reward)
• Tool usage attempts even on incorrect answers (Tool Call Reward)
• Concise responses (overlong penalty)

Training Pipeline

Qwen3-4B-Instruct-2507
  │
  ├── Stage 1: SFT (multi-turn agentic cold-start)
  │     Dataset: y-ohtani/open_agentrl_like_sft (2K samples, Apache-2.0)
  │     Epochs: 10, Max length: 32768, Full fine-tuning (FSDP, bfloat16)
  │
  └── Stage 2: GRPO-TCR (this model)
        Dataset: y-ohtani/open_agentrl_grpo_2k (8 samples = test run, Apache-2.0)
        Epochs: 3, Total steps: 6, Algorithm: GRPO + 5 enhancements (see below)

GRPO-TCR Configuration

5 Key Enhancements over Standard GRPO

Validation Results (Step 6)

Dataset (RL Stage)

• Name: y-ohtani/open_agentrl_grpo_2k
• License: Apache-2.0 (all sources are Apache-2.0 or MIT)
• Sampling: 8 samples (test run), balanced across 5 sources

All training data is sourced from Apache-2.0 / MIT licensed open datasets. This repository does NOT redistribute the dataset.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "y-ohtani/GRPO-TCR-Qwen3-4B-test"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Find all prime numbers p such that p^2 + 2 is also prime."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & Terms

Users must comply with the base model license and dataset terms.

Intended Use & Limitations

• Intended: Pipeline validation only. This is a test run to verify the training configuration.
• Not intended: Any production or evaluation use.
• Limitations:
  • Trained on only 8 samples (6 steps) as a pipeline validation run.
  • Tool calling requires a compatible runtime (e.g., SandboxFusion, code interpreter).