searchr1-repro-4b

[!NOTE] For full information, go check out the ORBIT paper here. This a naive reproduction of the Search-R1 model using Qwen3-4B as backbone trained using a single GRPO step.

<img src="https://huggingface.co/orbit-ai/orbit-4b-v0.1/resolve/main/orbit-with-name-logo.png" alt="Figure 1" width="500"/>

Reproduction of Search-R1 with Qwen3-4B base with NQ and HotpotQA (Step 200)

This is the RL-trained checkpoint of Search-R1, a small-sized (4B parameters) open search agent using the Qwen3-4B base model, fine-tuned with GRPO to use web search as a tool for multi-turn question answering. It was trained using the verl-tool framework with a live DDGS-based retriever.

Training was conducted for 200 GRPO steps on the complete Natural Questions (NQ) and HotpotQA datasets.

---

Model Details

---

Training Dataset

The model was trained on a mixed retrieval QA dataset with equal sampling across three tasks:

Dataset name: PeterJinGo/nq_hotpotqa_train Train batch size: 256 samples per step (n=8 rollouts per sample → 2048 trajectories/step)

---

Training Hyperparameters

---

Tool Configuration

The model was trained with live web search via a DDGS-based retrieval server:

The retriever server runs as a FastAPI service. At each agent turn the model issues a <search> query </search> action; the tool server retrieves results and returns them as <information>…</information> observations. The trajectory ends when the model emits <answer> … </answer> or the turn budget is exhausted.

---

Training Infrastructure

---

Reward Model

Reward manager: search_r1_qa_em
Reward is computed as exact-match (EM) between the model's <answer> and the reference target string. Multi-turn rollouts are scored at the final answer only; intermediate search turns receive no intermediate reward. Observations are masked from the KL and policy gradient loss.

---

Usage

This model is designed to be used with a running tool server that handles <search> actions. Inference without a live retriever will fall back to the model's parametric knowledge.

With verl-tool (recommended)

git clone https://github.com/TIGER-AI-Lab/verl-tool
cd verl-tool
uv sync
source .venv/bin/activate

# Start the DDGS retriever
python ddgs_retrieval_optimized.py --port 8280 --topk 5 \
    --backend "google,brave,bing,wikipedia,grokipedia"

# Start the tool server
python -m verl_tool.servers.serve \
    --host 0.0.0.0 --port 30500 \
    --tool_type search_retrieval \
    --workers_per_tool 4

Direct inference (parametric knowledge only)

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "orbit-ai/searchr1-repro-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

prompt = (
    "Answer the given question. Please break down the question, using it to plan "
    "a potential solution trajectory. You must conduct reasoning inside <think> and "
    "</think> first, then you may use tools to gather information. "
    "For search, use <search> query </search>. "
    "Provide your final answer with <answer> answer </answer>.\n\n"
    "Question: What percentage of blood is made up of plasma?"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=1.0, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

Intended Use & Limitations

• Intended use: Research into multi-turn retrieval-augmented reasoning and RL-based tool-use training.
• Language: English only.
• Search dependency: Peak performance requires a live web search backend. Without search, the model still reasons using parametric knowledge but accuracy degrades on fine-grained entity questions (particularly InfoSeek).
• Not intended for production deployment without additional safety filtering.

---

Citation

If you use this model or the training methodology, please cite:

@misc{thakur2026orbit,
      title={ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget}, 
      author={Nandan Thakur and Zijian Chen and Xueguang Ma and Jimmy Lin},
      year={2026},
      eprint={2604.01195},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.01195}, 
}

---