Downloads · 30 days
6
7% of all-time downloads
dalek-ai/baseline-v1-desc-hybrid-next-v1
baseline-v1-desc-hybrid-next-v1 is a sentence similarity model from dalek-ai. Use it when you need a score for how close two texts are. It is set up for agent-tool-router. The card lists the license as mit.
Same hybrid pipeline as baseline-v1-desc-hybrid, but the encoder has been fine-tuned on 8 386 next-tool prediction triplets with contrastive (task, golddescription, hardnegative) triples. Hard negatives are sampled fr…
Downloads · 30 days
6
7% of all-time downloads
All-time downloads
86
Public
Repo size
34.5 MB
Likes
0
Public
Click a slice to open those files.
.npy28.7 MB · 82%
From the Hugging Face model README
Same hybrid pipeline as
baseline-v1-desc-hybrid,
but the encoder has been fine-tuned on 8 386 next-tool prediction
triplets with contrastive (task, gold_description, hard_negative)
triples. Hard negatives are sampled from the top-50 retrieval of the
previous hybrid encoder, so each gradient step pushes the encoder
past its own retrieval mistakes.
Encoder lives at
dalek-ai/minilm-next-v1,
loaded lazily on the first route() call. ~30 MB of TF-IDF + encoder
centroids on disk in this repo.
Install the SDK directly from GitHub (PyPI publish pending):
pip install git+https://github.com/dalek-ai/agent-tool-router.git
pip install "agent-tool-router[encoder] @ git+https://github.com/dalek-ai/agent-tool-router.git"
from agent_tool_router import Router
r = Router.from_pretrained("baseline-v1-desc-hybrid-next-v1")
r.route(
"I want to add a checked bag to my reservation",
k=3,
history=["update_reservation_flights"],
)
Session 25 found that a Markov-1 rerank on top-200 retrieval saturated the mechanical ceiling (recall@200 = 69.6% on held-out next-tool triplets, Markov-1 reached 54.9% top-3 = 99% of that ceiling). To go past 55% top-3 the retriever itself had to improve.
Next-tool prediction on 2 094 held-out triplets (trace_id 80/20 split, seed=17, Markov-1 with α=0.4 rebuilt from the 80% train slice):
| Encoder | recall@50 | recall@200 | Markov top-3 K=50 | Markov top-3 K=200 |
|---|---|---|---|---|
MiniLM-L6 (baseline-v1-desc-hybrid) | 58.9% | 69.6% | 48.0% | 54.9% |
minilm-next-v1 (this model) | 83.4% | 93.1% | 70.2% | 75.5% |
+23.5pp recall@200, +20.6pp Markov top-3 K=200. Reproduce:
python -m router.eval.eval_next_tool_widen --cache-dir data/cache/next_tool_v1 --model dalek-ai/baseline-v1-desc-hybrid-next-v1.
Passing a 2-step history triggers stupid backoff to the bigram table
shipped in this repo (markov2_counts.npz + markov2_keys.npy, ~25 KB),
falling back to Markov-1 when the (prev2, prev1) pair is unseen in train.
Same held-out 2 094 triplets, K=200, best alpha per system:
| system | top-1 | top-3 | top-5 |
|---|---|---|---|
| Markov-1 only (≥ 0.3.0) | 52.9% | 75.5% | 81.1% |
| Markov-2 backoff (≥ 0.4.0) | 57.3% | 77.4% | 81.6% |
| delta | +4.4pp | +1.9pp | +0.5pp |
On the 775 test rows whose (prev2, prev1) bigram is seen in train,
top-1 jumps +14.9pp (61.4% → 76.3%). The headline gain lands on
tau-bench t≥3 (long-horizon agents): 84.8% → 87.9% top-3 (+3.1pp).
Reproduce:
python -m router.eval.eval_next_tool_markov2 --cache-dir data/cache/next_tool_v1.
LOSO refit hybrid (full 18 671-tool catalog, α=0.5, top-3 per call):
| held_out | n_calls | MiniLM-L6 | minilm-next-v1 | delta |
|---|---|---|---|---|
| Hermes function-calling-v1 | 4 376 | 72.3% | 73.6% | +1.3pp |
| ToolACE | 17 169 | 58.7% | 64.8% | +6.1pp |
| tau-bench | 8 880 | 11.1% | 38.8% | +27.7pp |
The fine-tune Pareto-dominates the default encoder on every held-out source. The tau-bench gain (+27.7pp) is the largest single-step improvement on that benchmark since the v0 → v1-desc switch.
The training triples are English-only, so a concern is that the encoder over-fits English and loses French alignment. Parallel n=50 EN/FR queries against the full 18 671-tool catalog (top-3 per call, no Markov rerank, hybrid α=0.5):
| model | EN top-3 | FR top-3 | EN top-5 | FR top-5 |
|---|---|---|---|---|
baseline-v1-desc-hybrid (default MiniLM-L6) | 82% | 26% | 90% | 30% |
| baseline-v1-desc-hybrid-next-v1 | 86% | 28% | 94% | 28% |
| baseline-v1-desc-hybrid-multilingual | 82% | 54% | 90% | 62% |
The fine-tune does not degrade French (+2pp top-3, within noise) and
adds +4pp top-3 on English over the default hybrid. For French
queries the multilingual encoder still dominates (+26pp top-3 vs this
model). Reproduce: python -m router.eval.eval_fr_pretrained.
github.com/dalek-ai/agent-tool-router · MIT. Live demo: dalek-ai/agent-tool-router-demo (gradio Space, FR/EN).