DatapressoRM_Lora_v1

Current Evaluation System Analysis

1. Trajectory Structure

The LLM agent generates trajectories using the following XML structure:

<think>reasoning content</think>
<tool_call>{"name": "tool_name", "parameters": {...}}</tool_call>
<result>execution result</result>
...
<think>final reasoning</think>
<answer>final answer</answer>

2. Clip Extraction Process

• Clips: Each <think> + <tool_call> + <result> sequence forms one clip
• Final Clip: <think> + <answer> sequence for the final output
• Tools Categories:
  • deepsearch: Information search tools
  • microsandbox: Code execution tools (sandbox_start, sandbox_stop, sandbox_run_code, etc.)
  • tavily: Web crawling tools (tavily-search, tavily-extract, etc.)
  • perform_web_task: Web interaction tools (search_google, go_to_url, etc.)

3. Current Evaluation Metrics

Clip-Level Evaluation

• Reasonableness Score (0.0-1.0) for regular clips:

  • 0.8-1.0: Excellent reasoning, optimal tool selection
  • 0.6-0.8: Good reasoning with minor issues
  • 0.4-0.6: Adequate but with notable issues
  • 0.2-0.4: Poor reasoning, questionable tool selection
  • 0.0-0.2: Very poor, inappropriate tools

• Correctness Score (0.0-1.0) for final clips:

  • 1.0: Perfect match with ground truth
  • 0.8-0.9: Mostly correct with minor differences
  • 0.6-0.8: Correct core but some inaccuracies
  • 0.4-0.6: Partially correct, missing details
  • 0.2-0.4: Incorrect but shows understanding
  • 0.0-0.2: Completely wrong

Category-Level Evaluation

Each category has specific metrics (all 0.0-1.0):

DeepSearch: Information Relevance, Tool Use Quality, Source Quality, Information Synthesis MicroSandbox: Code Correctness, Tool Use Quality, Computational Efficiency, Result Interpretation Perform Web Task: Tool Use Quality, Content Extraction, Interaction Quality, Goal Achievement Tavily: Tool Use Quality, Information Relevance, Content Extraction, Goal Achievement Final Assessment: Task Completion, Tool Use Quality, Reasoning Coherence, Problem Resolution, Answer Correctness

Reward Model Integration Architecture

1. Reward Model Setup

Model Configuration

• Base Model: Fine-tuned Qwen-8B
• Input Format: Task instruction + previous clips summary + current clip content
• Output Format: JSON with scores and summaries
• Inference Mode: Local deployment for real-time evaluation

Reward Model Integration Architecture

Reward Model Setup

Model Configuration

• Base Model: Fine-tuned Qwen-8B
• Input Format: Task instruction + previous clips summary + current clip content
• Output Format: JSON with scores and summaries
• Inference Mode: Local deployment for real-time evaluation

Reward Design Schemes 以下可以做对比实验（或者说是消融实验）

Scheme 1: Output-Only Reward

This scheme focuses solely on the final output quality without considering the process.就是tool-star 那一套

Scheme 2: Combined Process + Output Reward (Primary Scheme)

将PRM 和 Output-Reward 结合

This is the core scheme that combines process evaluation with output evaluation, providing a balanced signal that values both the reasoning journey and the final destination.

数学公式

The combined reward scheme is mathematically formulated as:

$$R_{final} = \begin{cases} -1.0 & \text{if } \text{format_invalid}(T) \ \alpha \cdot R_{output} + \beta \cdot R_{process} & \text{otherwise} \end{cases}$$

Where:

• $\alpha = 0.7$ (output weight) - emphasizes final correctness
• $\beta = 0.3$ (process weight) - encourages good reasoning
• $\alpha + \beta = 1.0$ (normalized weights)
• 具体的$\alpha， \beta$参数可以结合实验调整

Output Reward Component

$$R_{output} = \begin{cases} 1.0 & \text{if } \text{format_valid}(T) \land \text{answer_correct}(T, G) \ 0.0 & \text{if } \text{format_valid}(T) \land \neg \text{answer_correct}(T, G) \ -1.0 & \text{if } \neg \text{format_valid}(T) \end{cases}$$

Where:

• $T$ = trajectory
• $G$ = ground truth
• $\text{format_valid}(T)$ = boolean function checking XML format compliance
• $\text{answer_correct}(T, G)$ = semantic similarity function $> 0.8$

Process Reward Component

The process reward integrates both clip-level and category-level evaluations:

$$R_{process} = \gamma \cdot R_{clips} + (1-\gamma) \cdot R_{categories}$$

Where $\gamma = 0.6$ balances clip and category contributions.

clip score 主要注重推理的逻辑性是否合理，category score 主要注重工具调用的质量如何

Clip-Level Process Reward: $$R_{clips} = \frac{\sum_{i=1}^{n} w_i \cdot s_i}{\sum_{i=1}^{n} w_i}$$

Where:

• $n$ = number of clips
• $w_i$ = progressive weight for clip $i$: $w_i = \frac{i}{n} \times \begin{cases} 1.5 & \text{if final clip} \ 1.0 & \text{otherwise} \end{cases}$
• $s_i$ = clip score: $\begin{cases} \text{correctness_score}_i & \text{if final clip} \ \text{reasonableness_score}_i & \text{otherwise} \end{cases}$

Category-Level Process Reward: $$R_{categories} = \frac{\sum_{k=1}^{m} \lambda_k \cdot \bar{s}k}{\sum{k=1}^{m} \lambda_k}$$

Where:

• $m$ = number of categories
• $\lambda_k$ = category importance weight
• $\bar{s}k$ = average metric score for category $k$: $\bar{s}k = \frac{1}{|M_k|} \sum{j \in M_k} s{k,j}$
• $M_k$ = set of metrics for category $k$

Progressive Weighting Rationale

The progressive weighting scheme $w_i = \frac{i}{n}$ ensures that:

1. Early clips (foundation setting) receive lower weights
2. Later clips (building on context) receive higher weights
3. Final clip receives a 1.5× boost for task completion importance

This reflects the natural importance hierarchy in multi-step reasoning tasks.

Scheme 3: Format Penalty Override 格式奖励

This ensures that format violations receive immediate negative rewards regardless of other factors.

Implementation

def apply_format_penalty_override(base_reward: float, output_format_valid: bool) -> float:
    """
    Override any positive reward if format is invalid
    """
    if not output_format_valid:
        return -1.0
    return base_reward