qwen2.5-1.5b-toolcalling-dpo

Qwen2.5-1.5B Tool Calling DPO

A DPO fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct, trained on NVIDIA's When2Call preference dataset to improve tool-calling decision making.

The model is trained to better distinguish between requests that require a tool call, requests that can be answered directly, and requests that cannot be answered using the available tools.

This model is based on Qwen2.5-1.5B-Instruct, which is licensed under Apache 2.0. The model was fine-tuned using NVIDIA's When2Call dataset, which is licensed under CC BY 4.0. The dataset license and attribution requirements apply to the use of the When2Call dataset.

Model Details

The model was initially fine-tuned using LoRA and Direct Preference Optimization (DPO). The LoRA adapter was subsequently merged into the base model using merge_and_unload() to produce a standalone model.

Dataset

Training was performed using the train_pref split of NVIDIA's When2Call dataset.

When2Call is designed specifically to evaluate and train LLMs on decisions about when (and when not) to call tools. The dataset includes preference pairs consisting of a chosen and rejected response for a given user request and tool specification.

The train_pref split contains 9,000 preference-training examples with:

• Tool specifications
• User messages
• Chosen responses
• Rejected responses

NVIDIA provides both an SFT dataset (train_sft) and a preference dataset (train_pref); this model uses the preference dataset for DPO training.

The When2Call dataset is synthetic and is licensed under CC BY 4.0.

Training Objective

The objective was to improve the model's tool-use decision boundary.

The model learns to distinguish between:

1. Tool Call — a tool should be invoked to answer the request.
2. Request for Information — the request can be handled without invoking a tool.
3. Cannot Answer — the available tools cannot answer the request.

The preference-training setup encourages the model to prefer appropriate responses over incorrect tool-calling behavior.

Evaluation

The fine-tuned model was evaluated against the original Qwen2.5-1.5B-Instruct model on 300 samples.

The evaluation uses the same 300-sample LLM-as-a-judge subset provided by When2Call. NVIDIA's dataset contains a larger 3,652-sample MCQ test set and a 300-sample LLM-as-a-judge subset.

Results

Key Results

The largest improvement was in unsupported tool calls:

45.0% → 4.3%

This indicates that DPO substantially reduced inappropriate or hallucinated tool invocations.

Intent accuracy also increased:

52.7% → 74.0%

and tool precision increased:

40.8% → 72.9%

However, this came with a substantial reduction in tool recall:

93.0% → 35.0%

and Tool F1:

56.7% → 47.3%

Therefore, the main behavioral change is a shift toward a more conservative, precision-oriented tool-calling policy.

Confusion Matrix

Qwen2.5-1.5B-Instruct

Qwen2.5-1.5B + DPO

The confusion matrix shows that DPO significantly reduced the model's tendency to issue tool calls for requests that should not result in a tool invocation.

Tool Calling Example

Tool Definition

tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

User

Can you check Nvidia's current stock price?

Model Output

<tool_call>
{"name": "get_stock_price", "arguments": {"ticker": "NVDA"}}
</tool_call>

Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

model.eval()

tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

messages = [
    {
        "role": "user",
        "content": "Can you check Nvidia's current stock price?"
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False
)

print(response)

Output Post-Processing

During evaluation, model outputs were passed through a lightweight post-processing step to normalize tool-call formatting.

The post-processing:

1. Extracts JSON containing name and arguments.
2. Removes duplicate or nested <tool_call> wrappers.
3. Normalizes the output to:

<tool_call>
{"name": "...", "arguments": {...}}
</tool_call>

If no tool-call JSON is detected, the output is treated as a normal conversational response.

The post-processing step is used for format normalization and evaluation and does not generate a tool call that the model did not produce.

Limitations

The main limitation is the precision-recall trade-off introduced by DPO.

The model is considerably better at avoiding unsupported tool calls, but it also misses a larger proportion of valid tool-call opportunities.

Therefore, this model should not be interpreted as universally better than the base model for tool calling. Instead, it demonstrates that preference optimization can strongly shift the tool-use decision policy of a small instruction-tuned model.

The evaluation also uses a relatively small 300-sample subset, so additional evaluation on larger and more diverse tool-calling benchmarks would be useful.

Intended Use

This model is intended for research and experimentation involving:

• Tool calling
• Function calling
• Tool-selection policies
• Preference optimization
• DPO
• Agentic LLM systems
• Small language model alignment

It is not intended to be considered production-ready without additional task-specific evaluation.

Future Work

• Recover tool-call recall while maintaining low unsupported-call rates
• Experiment with DPO hyperparameters
• Improve preference-data construction
• Compare DPO against SFT
• Evaluate larger Qwen models
• Evaluate multi-tool selection
• Evaluate multi-step tool calling
• Improve argument-generation accuracy
• Benchmark inference using vLLM
• Evaluate on larger tool-calling benchmarks

Base Model

This model is based on:

Qwen/Qwen2.5-1.5B-Instruct

Please refer to the base model for its original capabilities, license, and usage restrictions.

Dataset Citation

This work uses NVIDIA's When2Call dataset.

Ross, Hayley, Ameya Sunil Mahabaleshwarka, and Yoshi Suhara. "When2Call: When (not) to Call Tools." NAACL 2025.

@inproceedings{ross-etal-2025-when2call,
    title = "{W}hen2{C}all: When (not) to Call Tools",
    author = "Ross, Hayley  and
      Mahabaleshwarkar, Ameya Sunil  and
      Suhara, Yoshi",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.174/",
    doi = "10.18653/v1/2025.naacl-long.174",
    pages = "3391--3409",
    ISBN = "979-8-89176-189-6",
    abstract = "Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling{---}whether the correct tool is called with the correct parameters{---}and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can{'}t be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts."
}

Acknowledgements

Thanks to the NVIDIA When2Call authors and the Qwen team for releasing the dataset and base model used in this experiment.