gen-router-t1

Gen Router Tier Classifier

The Problem

Every LLM-powered application has to decide, for each incoming prompt, which model should answer it. Get that decision wrong in either direction and it costs you:

• Always use the most capable model and you pay frontier-tier pricing and latency for questions a much cheaper model could have answered just as well — "what's the capital of France?" doesn't need a reasoning-heavy, expensive model.
• Always use a cheap, fast model and you save money on easy traffic but silently degrade quality on the hard prompts that actually need deep reasoning, domain expertise, or careful multi-step work — a failure mode that's easy to miss until a user notices a bad answer.
• Route by hand-written rules (keyword lists, prompt length, source of the request) and you get a system that's brittle and doesn't generalize — short prompts can be genuinely hard, and long prompts can be trivial.

This model exists to make that per-request decision automatically and cheaply: given a prompt, predict whether it belongs on a fast, tier, balanced tier, or frontier tier model — before any generation happens — so a routing layer can send each request to the right-sized model. Because the classifier itself is small (see benchmark below), the routing decision adds negligible cost and latency compared to running the prompt through an oversized model unnecessarily.

What It Predicts

Test Set Results

Label Mapping

{
  "0": "fast",
  "1": "balanced",
  "2": "frontier"
}

Training

• 4 epochs, lr=2e-5, batch size 32, max_length=320
• Class-weighted cross-entropy loss (inverse frequency) to counter frontier being a minority class
• Best checkpoint selected by validation F1 macro (monotonically improved every epoch: 0.708 → 0.788 → 0.792 → 0.798)

Comprehensive Benchmark

Per-Source Accuracy (test set)

Calibration

Expected Calibration Error (ECE): 0.053.

Inference Latency — PyTorch vs ONNX (single-sample, batch=1)

The ONNX export runs ~3x faster on GPU and ~2x faster on CPU than the native PyTorch model, with accuracy verified identical on the full test set (89.02% for both). CPU tail latency (p95/p99) reflects a constrained benchmarking environment and is not representative of a dedicated server.

Error Analysis

• Overall accuracy: 89.0% (test set, n=13,947)
• Critical failure rate (frontier prompt routed to fast): 3.6%
• Over-provisioning rate (fast prompt routed to frontier): 0.17%

The error asymmetry favors safety: the model is far more likely to over-provision an easy prompt (wasting some cost) than under-provision a hard one (risking a bad answer) — the direction you want a router to err in.

Usage (PyTorch)

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)

inputs = tokenizer("Explain quantum entanglement in detail.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
tier = model.config.id2label[logits.argmax(-1).item()]
print(tier)  # "frontier"

Usage (ONNX — faster, recommended for production)

import onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
onnx_path = hf_hub_download(repo_id, "onnx/model.onnx")
session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

enc = tokenizer("Explain quantum entanglement in detail.", return_tensors="np")
inputs = {i.name: enc[i.name] for i in session.get_inputs()}
logits = session.run(["logits"], inputs)[0]