toolgrad-4b

ToolGrad 4B

ToolGrad 4B is a fine-tuned version of google/gemma-3-4b-it optimized for function calling and tool-use tasks. It is trained on the dataset generated using the method described in our paper ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients" (ACL 2026 Finding). The codebase is available at our GitHub Repository.

Model Details

• Developed by: Zhongyi Zhou
• Model Type: Causal Language Model
• Base Model: google/gemma-3-4b-it
• License: gemma-terms-of-use

Intended Use

Single-turn tool-use tasks.

Evaluation Results

Evaluated on the Berkeley Function Calling Leaderboard (BFCL) v1 & v2:

Overall & Hallucination Scores

<table style="border-collapse: collapse;"> <thead> <tr> <th rowspan="2" style="border-bottom: 1px solid #ddd; padding: 8px;">Model</th> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Non-live</th> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Live</th> <th colspan="2" align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Halluc.</th> </tr> <tr> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Overall</th> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Overall</th> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Rel.</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">Irrel.</th> </tr> </thead> <tbody> <tr> <td style="padding: 8px;"><b>Gemma-3 4B</b></td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">61.12%</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">60.84%</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">53.94%</td> <td align="center" style="padding: 8px;">100.00%</td> </tr> <tr> <td style="padding: 8px;"><b>ToolGrad 4B</b></td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">72.46% ↑</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">65.58% ↑</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">93.75%</td> <td align="center" style="padding: 8px;">59.27%</td> </tr> </tbody> </table>

Detailed Category Scores

<table style="border-collapse: collapse;"> <thead> <tr> <th rowspan="2" style="border-bottom: 1px solid #ddd; padding: 8px;">Model</th> <th colspan="4" align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Non-live</th> <th colspan="4" align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Live</th> </tr> <tr> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Simple</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">Multi</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">Par</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">MultiPar</th> <th align="center" style="border-left: 1px solid #ddd; border-bottom: 1px solid #ddd; padding: 8px;">Simple</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">Multi</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">Par</th> <th align="center" style="border-bottom: 1px solid #ddd; padding: 8px;">MultiPar</th> </tr> </thead> <tbody> <tr> <td style="padding: 8px;"><b>Gemma-3 4B</b></td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">64.50%</td> <td align="center" style="padding: 8px;">88.00%</td> <td align="center" style="padding: 8px;">56.00%</td> <td align="center" style="padding: 8px;">36.00%</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">70.93%</td> <td align="center" style="padding: 8px;">59.35%</td> <td align="center" style="padding: 8px;">25.00%</td> <td align="center" style="padding: 8px;">41.67%</td> </tr> <tr> <td style="padding: 8px;"><b>ToolGrad 4B</b></td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">65.33% ↑</td> <td align="center" style="padding: 8px;">86.50% ↓</td> <td align="center" style="padding: 8px;">73.00% ↑</td> <td align="center" style="padding: 8px;">65.00% ↑</td> <td align="center" style="border-left: 1px solid #ddd; padding: 8px;">71.32% ↑</td> <td align="center" style="padding: 8px;">64.86% ↑</td> <td align="center" style="padding: 8px;">43.75% ↑</td> <td align="center" style="padding: 8px;">50.00% ↑</td> </tr> </tbody> </table>

How to Get Started

You can load this model using the transformers library:

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "zhongyi-zhou/toolgrad-4b"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Citation

If you find this work helpful, please cite our paper:

@misc{zhou2026toolgradefficienttoolusedataset,
      title={ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"}, 
      author={Zhongyi Zhou and Kohei Uehara and Haoyu Zhang and Jingtao Zhou and Lin Gu and Ruofei Du and Zheng Xu and Tatsuya Harada},
      year={2026},
      eprint={2508.04086},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2508.04086}, 
}