gavel-decide-4b

Gavel-Decide 4B — pairwise decision head on a frozen Qwen3-4B

A small MLP scoring head over frozen Qwen3-4B-Base hiddens. One forward pass per (state, option) pair; a softmax with a fitted temperature turns the per-option scores into a distribution; argmax decides. No text is generated, so output_tokens is always 0. Answers choice, noul and score.

The trunk is never fine-tuned. Only the head is trained, and it trains in seconds on CPU.

Headline: 74.46% on the 231 public JevBench items

Calibration axis 87.15 measured out-of-fold (binned ECE 0.0642, n=137), against 91.64 in-sample on the fitted set. A nested estimate, refitting the calibrator inside each fold, lands nearer 86.06.

Reproduce with scripts/verify_claims.py and scripts/verify_calibration_invariants.py in the submission repo; every number above is asserted against a file in results/.

Read this before quoting the accuracy: 213 of 231 items are training data

The head was trained on 689 option pairs drawn from 213 of the 231 public JevBench items. Split by whether the item was in that training set:

So 74.46% is not a held-out number, and neither is 79.34%. Only 18 items are genuinely held out, and the model gets 3 of them right — barely below the 0.318 chance rate. Treat the accuracy above as a fit statistic, not as generalisation.

Two independent checks on whether the head learned anything transferable:

• A head trained on NLI only (860 items, zero JevBench items) scores 0.5285 out-of-fold on NLI and 2/12 = 16.7% on JevBench items it never saw — also near chance. The NLI task is a different mapping, not a substitute for JevBench.
• The 18 held-out items are the only clean measurement here, and 3/18 is too small a sample to support a strong claim in either direction.

What has changed from earlier versions of this card

Previous revisions of this card reported 81.5% overall and a v0 table showing up to 90.5%. Those numbers came from evaluating only the 213 items in the training set (denominators 48 / 60 / 105), while being labelled overall_public. The 18 held-out items were not in the table. The current figures replace them: same weights, measured over all 231 items, with the contaminated/held-out split stated above.

The v0 table's 90.5% was additionally a pair-level figure (665/1073 pairs = 61.98%), not item accuracy. Pair and item accuracy are not comparable and were presented in one table.

Files

v1/combined_head.pt is byte-identical to the head used for the 74.46% run above; scripts/verify_hub_head.py checks the tensors and prints a SHA-256 of the pair.

Calibration

Two stages, both temperature-style, both argmax-preserving:

1. A fitted temperature over the option scores (T=2.4453).
2. A top-label meta-rescale: the top probability is predicted from four features (confidence, margin, entropy, option count) and the remaining mass is rescaled proportionally. Because the leader cannot change, the argmax is preserved by construction. scripts/verify_calibration_invariants.py checks this on all 231 shipped rows and reports 0 flips.

The temperature and the meta-rescale are fit on the same distribution the head is scored on, so the calibration figures above are optimistic. The 87.15 out-of-fold number is the defensible one.

Limits

• Context is truncated to 512 tokens. The head is trained on short contexts. Longer states are cut, not modelled. This matters for the Decision Index suite, whose items include multi-thousand-token HoVer and RAGTruth contexts; under that benchmark's rules an engine must either handle the full request or declare it unsupported, and cutting the state silently scores the wrong thing.
• Not RLCD. The trunk was never updated, and no RL objective is involved.
• The score type is handled, but the ordinal head was only partially validated.
• No Decision Index score has been measured. This model has not been run through the frozen 0.2.1 suite.
• The 18 held-out items are too few for a reliable accuracy estimate.

Reproducing the 74.46%

The trunk is public (Qwen/Qwen3-4B-Base, Apache-2.0). Cache hiddens for the 231 public items with the adapter in the submission repo, then score:

python scripts/run_jevbench_gavel.py --out results/run.json
python scripts/verify_claims.py

verify_claims.py recomputes accuracy, per-tier accuracy, the calibration figures and the contamination overlap from the raw run, and fails on any mismatch rather than printing a number that has drifted.

License

Head weights: MIT. The trunk Qwen3-4B-Base is Apache-2.0 and is not redistributed here. JevBench items are used as evaluation data under their own terms and are not redistributed in this repo.