aco-specialists-tier-router-v2

narcolepticchicken/aco-specialists-tier-router-v2

Tier router for Agent Cost Optimizer — classifies agent queries into 3 difficulty tiers (easy/medium/hard) to select the cheapest adequate model.

Model

• Base: ModernBERT-base (answerdotai/ModernBERT-base, 149.6M params)
• Architecture: ModernBertForSequenceClassification (3-class)
• Training: Focal loss (γ=2.0, α=[0.641, 1.044, 1.315]), 10 epochs, early stopping, dropout=0.1 throughout

Verified Metrics (independent evaluation, 2026-06-03)

Evaluated on 1,260 held-out RouterArena instances (class dist: easy=593, medium=359, hard=308).

Per-class F1: easy=0.768, medium=0.544, hard=0.720

Assessment: v2 offers no meaningful improvement over v1 for tier routing. The architecture upgrade (67M→149M) and focal loss did not help on this task. The model is usable but not better.

Training Data

RouteWorks/RouterArena (8,400 questions from 9 domains, 3 difficulty levels). Preprocessed split available at narcolepticchicken/aco-traces (data/tier_router/).

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "narcolepticchicken/aco-specialists-tier-router-v2")
tokenizer = AutoTokenizer.from_pretrained(
    "narcolepticchicken/aco-specialists-tier-router-v2")

inputs = tokenizer("Write a Python function to reverse a linked list", 
                    return_tensors="pt", truncation=True, max_length=2048)
outputs = model(**inputs)
# 0=easy, 1=medium, 2=hard

Limitations

• Medium-difficulty queries have poor recall (0.555) — the model over-assigns "easy" and "hard"
• No improvement over DistilBERT v1; prefer the smaller model for deployment
• Trained on academic QA benchmarks; may not generalize to coding/agent tasks