aco-specialists-tool-gater-v2

narcolepticchicken/aco-specialists-tool-gater-v2

Tool-use gater for Agent Cost Optimizer — predicts whether a user query requires tool calls. Only ~6% of agent queries actually need tools; the model should gate the rest.

Model

• Base: ModernBERT-base (answerdotai/ModernBERT-base, 149.6M params)
• Architecture: ModernBertForSequenceClassification (2-class)
• Training: Focal loss (γ=2.0, α=[0.111, 1.889]), 10 epochs, early stopping, dropout=0.1 throughout

Verified Metrics (independent evaluation, 2026-06-03)

Evaluated on 1,695 held-out ToolACE instances (class dist: no_tool=1,592, call_tool=103; 5.6% positive).

Per-class: neg P=0.977 R=0.934 F1=0.955 | pos P=0.393 R=0.660 F1=0.493

Assessment: v2 is significantly worse than v1. ModernBERT-base overfits on the 5.6% minority class despite focal loss and class weighting. The tool-gater-v2 should NOT be used; prefer v1 (DistilBERT).

Root Cause

The 5.6% positive rate (103 examples in test) is too few for a 149M-param model. ModernBERT's higher capacity + 2048-token context overfits on the majority class noise. DistilBERT's lower capacity and 512-token limit were actually beneficial as a regularizer.

Training Data

lockon/ToolACE (11,300 agent conversations with tool calls detected by regex). Preprocessed split available at narcolepticchicken/aco-traces (data/tool_gater/).

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "narcolepticchicken/aco-specialists-tool-gater-v2")
tokenizer = AutoTokenizer.from_pretrained(
    "narcolepticchicken/aco-specialists-tool-gater-v2")

inputs = tokenizer("Query: What is the capital of France?",
                    return_tensors="pt", truncation=True, max_length=2048)
outputs = model(**inputs)
# 0=no_tool, 1=call_tool

Limitations

• Regression vs v1: Use DistilBERT v1 (narcolepticchicken/aco-specialists-tool-gater) instead
• Poor precision (0.393) on the "call_tool" class means many false positives
• Extreme class imbalance (5.6%) requires more training data or data augmentation