rsi-jev-v1.0-qwen3.5-2b

rsi-jev-v1.0-qwen3.5-2b

A typed-decision model: Qwen/Qwen3.5-2B-Base with its tower fine-tuned and a trained option-scoring head. It answers choice / noul / score questions about a document in one forward pass, and speaks the Jev HTTP API.

This checkpoint is seed 17, the seed fixed in advance as primary, not the best of the three:

Re-computed from these exact files after upload-time reload. Kernel stack fla-0.5.2/torch-2.7.1+cu128; full recipe in meta.json.

import sys; sys.path.insert(0, 'code')   # after snapshot_download of this repo
from load_release import load_release
model, tok, enc, meta = load_release('.', device='cuda')

These models are trained by an autonomous research loop, so telling us what is wrong with one is the most useful thing you can do. A case where it is confidently wrong becomes a test case and often a hypothesis; a variant you tried that failed removes a branch nobody has to pay for again. Either one belongs in this repository's discussions.

Everything below is this release's own record, unchanged.

The code, this release's record, and every experiment behind it live at github.com/Shanghua-Gao/RSI-Jev.

These models are trained by an autonomous research loop, so telling us what is wrong with one is the most useful thing you can do. A case where it is confidently wrong becomes a test case and often a hypothesis; a variant you tried that failed removes a branch nobody has to pay for again. Either one belongs in this repository's discussions.

Everything below is this release's own record, unchanged.

---

RSI-Jev v1.0

Describes release v1.0 · updated 2026-09-25

Two typed-decision models, 0.8B and 2B. Given a document and a set of typed questions — yes/no, pick-one-of-k, rate-on-a-rubric — they return a calibrated probability for every option in one forward pass, with no generated text.

On the LocalLLaMA/typed-decisions test set — 400 documents, 2,000 decisions — the 2B model scores 0.662 pooled top-1 and the 0.8B 0.614, against a majority baseline of 0.5185 and Jev's 0.727.

To download and use a checkpoint, see the root README; for the HTTP API, ../serve/README.md. This card is the evidence behind both.

---

1. The task

A typed decision is a document (the state) plus one or more questions, each of a declared type:

Pooled top-1 is the fraction of all 2,000 test decisions whose highest-probability option matches the teacher's modal label. It is the headline throughout, because a model can look good on one question type and fail another.

2. How the model is built and trained

The scorer is a cross-attention head: the hidden state at the decision cue is the query, the mean-pooled option spans are keys and values, and the attended result projects to one logit per option. Because it attends over the whole option set, an option's logit depends on what it competes against.

Four of those rows were forced by a failure. Fine-tuning the tower rather than freezing it, because every frozen variant stalled below the majority baseline whatever the head or the data. The scorer in fp32 outside autocast, which took seven failed experiments to establish. Options shuffled per example, because a causal encoder shows option k only options 1…k−1, and a head trained on a fixed order collapses onto position — one early run picked the last option on 800 of 800 score questions. Three seeds, because the sd across them is 0.011 and single-seed differences smaller than that are noise. All of it is in §10.

The benchmark is held out completely, verified rather than asserted: no training document shares a state with the benchmark's train or test split, and none shares even a single 12-word phrase with either.

Full spec in each checkpoint's meta.json; code and a reproduction recipe in ../rsijev/README.md.

3. Checkpoints

Two benchmarks, not three: the middle columns are the same 2,000 questions scored twice, once in the dataset's option order and once reversed. A model answering by position would score differently on the two; the gap here is at most 0.0035. MMLU-Pro is a separate guard metric (§4). All figures pooled top-1. Apache-2.0, following their base models.

4. Results

Three seeds, full test set:

Against the reference points that give those numbers meaning:

• Scale helps: +0.048 from 0.8B to 2B on typed-decisions, +0.10 on MMLU-Pro.
• MMLU-Pro improves rather than degrades — 0.186 → 0.269 at 0.8B, 0.315 → 0.355 at 2B. It is a guard metric, checked so that gains on decisions are not bought by damaging general ability.
• Calibration improves sharply: score-mode Brier 0.111 at 2B against 0.335 for the untrained readout.
• Option-order sensitivity shrinks to at most 0.012 at 2B between canonical and reversed presentation, from 0.036 in earlier attempts.

Where the errors are. Pooled top-1 hides that the benchmark's own teacher is unsure about most of it: 61% of test items have a gold max-probability below 0.67 and carry 75% of the errors. Scored only where the teacher was confident, the 2B model reads 0.784 at gold max-prob ≥ 0.67 and 0.886 at ≥ 0.9. It fits the synthetic teacher it was trained on at 0.850 on held-out synth against 0.662 on the benchmark, so the gap is teacher mismatch rather than underfitting. A useful reference point measured alongside: zero-shot Qwen3.5-4B with a 1,024-token thinking budget scores 0.677 here and 0.964 on the items its teacher was sure of — usable at training time only, since Jev is System One and generates nothing.

Verification. Each released checkpoint was reloaded from disk and re-scored; all 3,000 of its per-question predictions matched the training run that produced it — agreement 1.0000 for both. A checkpoint that does not reproduce its own run is not published. The released checkpoints are fresh runs at seed 17, so the 2B one reads 0.6525 where the experiment that set the headline read 0.6515: two questions out of 2,000, from non-deterministic GPU kernels. Both are reported rather than the friendlier one. Served end to end on GPU, a four-question request including a 64-option choice returns well-formed answers; the worked example and the answer it measurably returns are in ../serve/README.md.

5. Speed

Sections 5 to 7 are measurements of these checkpoints as served. Each table names the script that prints it, so any of them can be re-run. Everything is an NVIDIA DGX Spark (NVIDIA GB10, 122 GiB unified) unless another machine is named, and all of it moves with GPU load — up to ±20% between runs.

python scripts/bench.py --model 2b, bf16, median of ten, asking one document 1, 2, 4, 8, 16 and 32 questions. A single per-decision figure would hide the shape of it: reading the document is paid once, and each decision after that is nearly free.

The fixed half grows with the document; the marginal cost of one more decision stays near 10 ms whatever the document. That is the whole argument for asking a batch: thirty-two decisions about one 404-token document take 418 ms together against 2,774 ms one request at a time, because the document is encoded once and every question continues from that cache. Four questions on those same three documents cost 108 / 148 / 241 ms at 2B and 89 / 125 / 198 ms at 0.8B — the smaller model is about 1.2x faster, which turns out not to be enough to build a cascade on (below).

Both versions stay resident at once in 5.1 GiB of 122 (1.45 + 3.61), so model size is not the constraint here, and all of it is on the plain PyTorch fallback rather than fla's fused kernels — a floor rather than a ceiling. For contrast, an A100 MIG 3g.20gb answers the four-question requests in 90 / 93 / 119 ms at 2B.

6. Acting on the probability

Speed is worth little if a confident answer is wrong. python scripts/calibration.py bins all 2,000 held-out decisions at 2B by the probability the model gave them, none of it seen in training:

Every bin lands at or above its stated confidence, so the model is conservative rather than overconfident — the useful direction for anything routing on a threshold — and zero of 2,000 decisions were confidently wrong (90%+ and incorrect). Overall accuracy is 65.1% against 51.9% for always guessing the majority. The top bin holds only 29 decisions and pins down little on its own; the 413- and 697-decision bins are the ones carrying the claim, and a bin's count moves by one or two between runs as a decision crosses a boundary.

That is worth something only if you use it. python scripts/routing.py ranks every decision by the probability behind it and keeps the top slice — the question a deployment actually asks, which accuracy alone cannot answer:

Strictly monotone: act on everything and 65% are right, act on the 40% it is surest about and 80% are right, with the rest going to a person. That trade exists only because the probability means something. The sweep is 2,000 decisions in 65 seconds on the Spark, both models resident, nothing generated.

The same script tries the obvious next idea — let the 0.8B answer and escalate what it is unsure about to the 2B — and it loses to simply running the 2B, which costs only 1.17x the 0.8B here, nowhere near the 5–10x a cascade needs. Repeat runs move the routing rows by up to 0.2 points — non-deterministic GPU kernels, not a different model.

7. What it costs to run

The scorer always runs in fp32 — that is the whole lesson of §10 — so there is no quantised path yet.

8. Fixed after release

Both are serving-code changes. The weights are unchanged, and so is every accuracy and calibration figure above.

The prefix cache was gated on a constant measured on an A100 with fused kernels. Break-even is a property of the machine, not the model: on the GB10's plain torch path, caching won at every size measured (1.09x at 161 tokens to 3.24x at 1,052), and the constant was refusing a 2x speed-up at 404 tokens. The gate now depends on whether the fused kernels are present, with RSIJEV_MIN_SAVED_TOKENS to override. The symptom had been sitting in our own published table — a 404-token document timed slower than a 1,052-token one.

confidence was the wrong statistic — and this one changes what the API returns. The field on a choice or score answer was computed as 1 - H(p)/ln K, chosen when the statistic was undocumented. It is in fact the peak statistic, (K * p_max - 1) / (K - 1) — TypeSafe documents it for three options as "(3 x largest probability - 1) / 2". The two agree exactly at a uniform distribution and at one-hot and nowhere else, which is why 25 wire-contract tests passed: they pinned only those two points. The gap on real answers is large — a four-option choice peaking at 0.90 reads 0.869 correctly against 0.692 as shipped — so anything that thresholded on confidence before this fix was acting on the wrong number. Accuracy and calibration are unaffected because the numbers in this record rank by the probability of the chosen option, a different quantity that did not change. noul answers carry no confidence, which was always correct here and matches the reference.

9. Limitations

• 0.065 behind Jev on typed-decisions, and far behind on MMLU-Pro, where the gap is base-model capacity rather than readout; training a 2B model does not close it.
• Selection bias. Versions were compared on the test split itself, so this ladder is optimistic by an amount that grows with the number of experiments run.
• Order sensitivity is reduced, not solved — up to 0.012 at 2B.
• Option keys are visible to the model. Renaming a choice key can move the answer; the reference implementation hides keys and is invariant to this.
• Above ~0.75 would be suspect. The dataset card warns that scores much above that mean a model has learned the teacher's quirks rather than the task.
• Not scaled past 2B. 4B needs sharding or an 8-bit optimiser.

10. How it got here

Five attempts came before this one, none released. They are recorded because the release is only interpretable next to them.

Why the scorer runs in fp32

That last instability is where the release came from. Seven remedies were tried, each at two seeds against a stability rule fixed in advance: a cosine schedule on the head, head learning rates 5e-5 and 2.5e-5, parameter-free input normalisation, label smoothing 0.02, head weight decay 0.1. Every one reduced the logit spikes; none removed them.

They failed because each treated the optimiser, and the optimiser was not the problem. Under bf16 autocast, nn.Linear and nn.MultiheadAttention compute in bf16 whatever dtype their weights are stored in. The scorer — documented, intended and reported as fp32 — had been computing its attention softmax, projections and gradients in bf16 in every attempt that trained the tower.

The evidence had been in the logs throughout: attempts with a frozen tower never spiked, and they run without autocast; every tower-training attempt could spike, and which seed did was luck; the head's weight norm moved 0.1% while its logits reached 12,096; and the largest input-feature norms reaching the head stayed flat. A head whose weights barely move cannot produce logits of 12,096 through optimisation. That mismatch is what pointed at precision.

Both columns are runs on the same kernel stack at the same head learning rate, which is the only way they are comparable. A consequence worth stating: head learning rate 1e-3 — the value that diverged at 0.8B and forced the reduction to 1e-4 — is stable once the scorer is fp32. Every learning-rate reduction on the way here had been compensating for the same bug.

Two failures worth more than the successes

A metric can certify a loser. On score, a head reading Spearman +46 had worse log score and Brier than the untrained control. Spearman is invariant to monotone reshaping, so it rewarded ordering while the probabilities themselves got worse. Score mode has been judged on proper scoring rules first ever since.

A readout can be provably blind and still look good. The score options " 0", " 1", " 2", " 3" all begin with the same space token, so a readout that looks only at the first token carries no option identity at all — and still returns a respectable-looking number. The control now reports first-token collisions, and any arm with collisions is invalid.