Downloads · 30 days
10
25% of all-time downloads
ymoslem/ModernBERT-base-AIME-24-25-26-router-qe-binary-vibethinker1.5b-5runs_1eval-10ep-lr5e-05
ModernBERT-base-AIME-24-25-26-router-qe-binary-vibethinker1.5b-5runs_1eval-10ep-lr5e-05 is a text classification model from ymoslem. Use it when you need a label for a piece of text. The card lists the license as cc-by-nc-sa-4.0.
A binary accept/route classifier for a two-tier LLM cascade on AIME. It reads a VibeThinker-1.5B answer and predicts whether that answer is correct. A route prediction means escalate the query to a stronger model.
Downloads · 30 days
10
25% of all-time downloads
All-time downloads
40
Public
Parameters
150M
598 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors598 MB · 99%
From the Hugging Face model README
A binary accept/route classifier for a two-tier LLM cascade on AIME. It reads a
VibeThinker-1.5B answer and predicts whether that answer is correct. A route
prediction means escalate the query to a stronger model.
Trained for CRE-Router on
ymoslem/AIME-24-25-26-router.
The generator is named on purpose. This estimator is trained on VibeThinker-1.5B outputs and should be applied to VibeThinker-1.5B outputs. It does not transfer to a different efficient tier without being retrained.
Build the input exactly as it was trained, or the scores will not mean what they should:
text = f"{question} {tokenizer.sep_token} {last_1000_words_of_output} "\
f"{tokenizer.sep_token} {num_output_tokens}"
Keep only the last 1,000 words of the model's output, then truncate at 4,096
tokens. Class 0 is route, class 1 is accept.
On the 450 rows of test_vibethinker, 90 problems from AIME 2024 to 2026 scored
across 5 sampled runs:
| Class | F1 | Precision | Recall |
|---|---|---|---|
| Route | 0.779 | 0.789 | 0.769 |
| Accept | 0.924 | 0.920 | 0.928 |
| Macro | 0.851 |
Accuracy 0.887. It escalates 114 of 450, catching 90 of the 117 wrong answers and wrongly escalating 24 correct ones.
The evaluation split is also the early-stopping split. This is deliberate, so the checkpoint stays comparable with the earlier estimators in this project, but it means 0.851 is a selection criterion rather than a held-out score.
Much of the signal is output length, and a length rule does as well. The input string ends with the token count, and the predicted accept probability correlates $-0.77$ with it. Composing the actual cascade, VibeThinker-1.5B escalating to Qwen3-30B-A3B-Thinking, and matching the number of escalations:
| Escalation rule | Escalated | Cascade accuracy |
|---|---|---|
| none, always VibeThinker | 0 | 74.0% |
| random | 115 | 76.7% |
| this classifier | 114 | 83.1% |
| longest answers first | 115 | 84.1% |
| all, always Qwen3-30B | 450 | 85.8% |
Selecting clearly beats not selecting: against random escalation of the same count the classifier gains 6.4 pp. But it does not beat sorting by output length, and the difference between them, $+0.98$ pp for the length rule, has a 95% interval of $[-0.98, +3.47]$ that spans zero. Report the length rule as the baseline, not random.
ModernBERT-base, 10 epochs with early stopping at patience 4, best epoch 3, learning rate 5e-5, effective batch 64 (16 with 4 accumulation steps), class weights for the 86% accept majority, on one A100.
CC BY-NC-SA 4.0, inherited from the training data, whose AIME 2026 source carries non-commercial share-alike terms. The base model is Apache-2.0.