five-lane-router-modernbert-large

Five-Lane Request Router (ModernBERT-large)

A fine-tuned ModernBERT-large classifier that decides what kind of answer a request needs, before any larger model runs:

chat · image · search · audio · video

It was trained to take over the routing step from a 9B generative model (an internal GPTQ build of a Qwen3.5-9B derivative on vLLM, thinking disabled) in a personal AI assistant, and runs as a TensorRT FP16 engine on an NVIDIA Jetson Orin Nano. As of release it is integrated but not yet serving live traffic; the 9B still routes production.

Results (one sealed blind test)

Both routers were run on the same 60 held-out cases, three times each (180 calls). The cases were written separately from the training data, sealed before training, leak-checked against it, and used once.

Read these carefully. This is one sealed set of 60 cases, not a production soak. The model is deterministic, so the three repeats agree; the fair reading is 60 of 60 distinct cases. For intuition only: if 60 cases were independent random draws from real traffic, zero errors would still be consistent with an error rate of up to about 5%. These cases were written to probe the lane boundaries, not sampled from traffic, so that figure is an illustration, not an estimate of the live error rate. Latency is measured end to end from a client on the same network, including tokenization, for each router as deployed: this model as a TensorRT engine on a Jetson Orin Nano, the 9B on an Intel Arc GPU server. The hardware differs, so this compares routing calls in one setup, not model speed. The PyTorch weights here will run at different speeds on your hardware.

An earlier candidate trained on less data scored 174/180 on a different sealed set and failed: it sent one chat request that mentioned drawing to image, on all three repeats. This model was trained after 253 contrast rows were added for that failure shape, and was then tested once on a fresh sealed set.

Public, rerunnable eval

On a frozen 60-row balanced authored challenge, the pinned router scored 58/60 versus 46/60 for a predeclared TF-IDF baseline. This public challenge was authored after training with knowledge of the label rules. It is a teaching and regression set, not an independent blind holdout or a sample of live traffic. Both model misses are search boundary cases and remain visible in the raw result and receipts. The eval tutorial walks through label rules, freeze and overlap checks, the baseline, scoring, failure analysis, and exact rerun steps. The sealed 60-case comparison above is a separate evidence set.

Usage

The frozen public eval loaded this model with Transformers 5.17.0. An earlier 4.57.6 runtime could not load the released TokenizersBackend tokenizer.

from transformers import pipeline

router = pipeline("text-classification", model="ericmey/five-lane-router-modernbert-large", top_k=None)
router.tokenizer.truncation_side = "left"      # keep the END of long conversations

scores = router("user: can you draw me a lighthouse at dusk?", truncation=True, max_length=512)[0]  # all five lanes, best first
print(scores[0]["label"], scores[0]["score"])   # e.g. image 0.99...

Input format matters. The model was trained on the last up-to-8 conversation turns, one per line, formatted role: text for user and assistant turns only, with the request being routed on the last line:

user: what should i wear saturday?
assistant: Maybe the blue shirt you liked?
user: yeah i guess. show me

Truncate from the left at 512 tokens, or long conversations lose the request itself.

Training

code/train_router.py is a portable version of the training script (CUDA, Apple MPS or CPU). code/train_modernbert.py is the exact script that produced these weights. code/evaluate_router.py runs a gated evaluation, and code/export_onnx.py exports to ONNX with a PyTorch-parity check. To serve it, code/serve_trt.py runs a TensorRT engine behind a small HTTP API (POST /v1/route, GET /health) on NVIDIA Jetson or any TensorRT host; code/verify_export.py checks PyTorch against ONNX on a dev set (same route, and every logit within 0.01), and optionally checks the running TensorRT server for route parity only: the same route and token count, not its logits or tokenizer hash; and code/run_gate.py collects predictions for a separate scorer. It sends a case file to a router endpoint and records each route and its latency. It does not pass or fail anything: it exits 0 even when the endpoint errors, recording the route as ERR, so score its output with a scorer for this output format ({case, rep, route, status, …} rows) before drawing a conclusion. evals/score.py does not read this format. The full walkthrough is in the accompanying blog post: Train Your Own Request Router.

Limitations

• Narrow by design. It picks a lane; it does not write a reply or a brief. In the intended deployment, a larger model still writes the short description that image/voice/video/search stages need, on those turns only.
• One domain. Synthetic, English, informal requests to a companion-style assistant. Expect degradation on other registers. Fine-tune on your own lanes and data.
• Lane boundaries are product decisions, encoded in the labels (see the dataset card). If your product treats "show me my cat" differently, this model will be wrong for you in exactly that place.
• Fail safe. Wire it with a fallback: if the classifier errors, times out, or returns an unexpected label, route that turn the old way.

License

Apache-2.0, matching the base model.