rerank-statute-en

Quanfire Statute Reranker — rerank-statute-en (cross-encoder, English central statutes)

✅ Measured across four seeded runs — 2026-08-24 · this checkpoint +53.7 %, the procedure spans +36 % to +77 %

The most useful thing on this card is not a single number. We trained this procedure four times on one evaluation, changing nothing but the seed, and the answer moved by 2.13×. Almost every reranker card quotes one run. Here are all of ours.

draw	seed	Recall@1	delta	95 % CI (paired, B=2000)	val loss (selection metric)
1 — published here	1001	0.1111	+0.0388 (+53.7 %)	[+0.0241, +0.0535]	0.2905 ← best
2	1002	0.1278	+0.0556 (+76.9 %)	[+0.0395, +0.0710]	0.3103 ← worst
3	1003	0.0984	+0.0261 (+36.1 %)	[+0.0127, +0.0408]	0.3006
4	1004	0.1124	+0.0402 (+55.6 %)	[+0.0254, +0.0569]	0.2941

Every interval excludes 0. All four ran on the identical evaluation — 1,494 held-out queries, a 1,205-passage pool, and a first-stage baseline of 0.07228916 in all four, identical to eight decimal places. That invariant is what shows the evaluation did not move underneath the draws: the bi-encoder is never trained, so drift there would have meant the comparison was not the same comparison.

The published checkpoint is draw 1 — chosen by validation loss, never by its test result. Validation is carved from training documents and never touches the evaluation set. Selecting on the best test result instead would have let us headline +76.9 %. We do not quote that figure and neither should you: it is one draw, and the honest summary of four is the range.

An awkward finding we are publishing rather than burying: validation loss does not predict test recall across these draws. The best validation loss produced the third-best test result, and the worst validation loss produced the best one. At n=4 that is suggestive rather than conclusive — but it means our selection rule buys a defensible protocol, not performance. Within a single run, checkpoint selection still beats saving the last step; across runs it buys nothing we can demonstrate.

The +47.2 % this card carried before is neither withdrawn nor corrected. It was an unseeded run of the same procedure on the same evaluation — same 1,494 queries, same 1,205 pool, same 0.0723 baseline — and it lands inside the seeded range. Read it as a fifth draw. The one thing it lacked is reproducibility: its training was never seeded, so its inputs cannot be regenerated. That is the substantive upgrade in these weights — a change in kind, not a better score. That revision stays published and readable at tag v2.0.0.

Stage	Recall@1	95 % CI (paired bootstrap, B=2000)
bi-encoder (multilingual-e5-small) retrieve only	0.0723	[0.0589, 0.0857]
+ this cross-encoder rerank (draw 1)	0.1111	[0.0959, 0.1272]

No held-out query comes from an Act that appears anywhere in training. Recall@100 ceiling 0.7557 — everything above that is out of the reranker's reach.

Two caveats still travel with these numbers.

1. Attribution. Two things changed at once between the null re-measure and this family of runs — 2.93× the training volume, and best-checkpoint selection instead of last-step. They landed together and cannot be separated; neither ablation was run.
2. Volume. These runs train on 77.1 % of the volume the withdrawn run used (26,252 rows against 34,052). No arithmetic between the two means anything, and in particular nothing here is "the cost of the leak" — the comparison is not volume-matched and never will be.

What to expect. From these weights, the delta in the table above. From retraining this procedure yourself, somewhere in +36 % to +77 %. The absolute numbers are low because the task is hard and the pool is document-isolated — the first-stage bi-encoder reaches only 7.2 % at rank 1. Judge the reranker by the delta and its interval, not by 0.1111.

A fifth draw was attempted and did not complete. Both of its permitted attempts stalled — the first before CUDA had even initialised, holding 23 MiB of card memory and drawing 9 W — and neither produced a number of any kind. It is recorded here because "one draw is missing" invites a reader to suspect an inconvenient result was dropped. There was no result to drop: this is missing data, not a filtered set, and the loss carries no information about what that draw would have measured.

<details> <summary><b>The record — the withdrawal (2026-08-21) and the null re-measure that preceded this.</b> Retained in full; the figures inside it must not be quoted.</summary>

🛑 Number withdrawn — 2026-08-21

The +63.8 % improvement this card reported is withdrawn. It was measured on a contaminated evaluation split and no replacement number exists yet.

This reranker's training script built its split with a plain shuffle — random.shuffle(rows), first 1,200 rows to eval, the rest to train — with no exclusion of any kind between the two sides. Replaying that exact split shows:

held-out queries (1,200)	share
whose positive passage text also appears in training	928 (77.3 %)
whose query text also appears in training	829 (69.1 %)
drawn from a source Act that also appears in training	1,200 (100 %)
genuinely unseen on both text and document	0 (0 %)

Every held-out query came from an Act the model had trained on, and three in four had their answer passage verbatim in the training set. A reranker measured that way is being asked to recognise text it has already seen.

What is not withdrawn. The weights are unchanged and the model is not being unpublished. The build lesson below — that form-separable negatives make a cross-encoder learn a query-independent shortcut, and that negatives must match the candidate distribution seen at inference — was diagnosed from a training collapse, not from this eval, and stands on its own.

What replaces it — nothing, so far. Updated 2026-08-23. The clean re-measure has now run, on a split that holds out whole Acts. It came back null:

1,494 queries, pool of 1,205	Recall@1	95 % CI
bi-encoder alone	0.0723	[0.0589, 0.0857]
+ this reranker	0.0823	[0.0689, 0.0971]
delta	+0.0100 (+13.9 %)	paired [−0.0027, +0.0228] — spans zero

The paired interval includes zero, so on an honest split we cannot distinguish this reranker from no reranker at all. It does not clear our own bar, which is that a shipped claim carries a paired CI excluding zero. This model still has no published effectiveness number, and now has a measured failure to produce one. Do not quote +63.8 % anywhere.

We are not claiming the leak caused this, and the earlier promise on this card that the two runs would be "comparable" was not kept. The pool sizes match (1,205 vs 1,182), but the training volumes do not: the clean run trained on 8,966 rows against the withdrawn run's ~34,052 — 26.3 %. Three separate things all push the clean number down and none of them can be separated from the leak or from each other:

1. The trainer filtered the corpus to rows carrying mined negatives — cutting it by 73 % — and then never used those negatives, because negatives are drawn from other rows' positives. The filter was made obsolete by the very fix that rescued the first training run, and nothing caught it.
2. The saved weights came from the run's final step, with no best-checkpoint tracking, and the last 1,000 steps ran at roughly 1.7× the epoch-mean loss. We measured a checkpoint we know was not the best one the run produced.
3. Learning rate was constant with an effective batch of one query.

So "the model is ineffective" and "we starved the model" both fit this evidence, and this card will not pretend to know which. A corrected run — full training volume and checkpoint selection on a validation slice carved from training documents, never the evaluation set — is in flight. If it produces a significant number, that number will not be presented as a replacement for +63.8 %. Different volume, different checkpoint discipline; it would stand on its own or not at all.

A correction of record: an earlier internal notice attributed this to the shared fine-tuning pipeline's split filter. That was wrong — this script never used that pipeline. The defect here was its own, and simpler.

</details>

A cross-encoder reranker for English central-statutory (bare-Act) text. It reads a query and a candidate passage together and returns a single relevance score, used to reorder the top-k of a bi-encoder retriever — the second stage of a retrieve-then-rerank pipeline. It is a fine-tune of intfloat/multilingual-e5-small (MIT) with a scalar relevance head (AutoModelForSequenceClassification, num_labels=1), max_length 256.

This is Quanfire's first reranker. Where a bi-encoder embeds query and passage independently and ranks by cosine, a cross-encoder attends across the pair — slower, but able to catch relevance a bi-encoder misses. It is a companion to quanfire-ai/embed-statute-en: that model retrieves, this one re-ranks what it retrieves.

• Framework & code: github.com/quanfire-ai/quanfire-multilingual-embedding (Apache-2.0)

• Weights licence: Apache-2.0. The model is ours; the training text is bare-Act statutory content, train-safe under Copyright Act §52(1)(q)(ii) for a non-reconstructive model (it emits a score, never the source text). The corpus is not redistributed.

• Internal run (these weights): statute-reranker-v2-fulldata · base e5-small + scalar head · listwise cross-entropy over 1 positive + 6 form-matched negatives · lr 2e-5, 2 epochs, max_length 256 (bf16, CUDA) · 26,252 training rows · checkpoint e1s25748 selected on a 600-row validation slice carved from training documents (val loss 0.2891 vs the final step's 0.3052). The earlier, unmeasured weights remain pinned at tag v1.0.0.

  Data seeding only, corrected 2026-08-23. This line previously read seed 0. That is true of the document split, the validation carve and the bootstrap, and false of training, which is what a reader takes it to mean. The training script shuffles the row order and draws each row's negatives from the bare module-level RNG, and never calls torch.manual_seed; the ranking head is newly initialised on every run. The training order, the negatives and the head are therefore drawn fresh from OS entropy each time. The consequence was a limitation and not a correction of any figure: +47.2 % was a single draw from a procedure whose run-to-run spread had never been measured. Our cross-lingual reranker is why we said so rather than assuming it was small — a same-arm redraw there, on byte-identical negatives, changed the sign of the result.

  RESOLVED 2026-08-24. That spread has now been measured, and this paragraph's promise is discharged rather than quietly dropped. Four seeded draws on one evaluation returned +36.1 %, +53.7 %, +55.6 % and +76.9 % — a 2.13× range in which every interval excludes zero. Two things follow. The direction is reproducible: no draw came back null, none changed sign, and nothing resembling the cross-lingual model's collapse appeared here. The magnitude is not: a single draw of this procedure cannot be quoted as its performance, which is why the top of this card leads with a range. The runner used for the spread also seeds training — random.seed, torch.manual_seed and torch.cuda.manual_seed_all, called after the split and before the head is built — so the defect described above is absent from the four draws, though it was present in the +47.2 % run they are compared against. Seeded is still not bit-reproducible: no deterministic-algorithm flag is set, and no same-seed re-run has been attempted.

What it is for

Re-ranking a retriever's shortlist on English central-statutory text. Given a query and the top-k passages a first-stage retriever returned, it scores each (query, passage) pair and sorts by score. It does not retrieve on its own — it needs a candidate set (typically a bi-encoder's top-50 or top-100). Its job is to lift the right section from somewhere in that shortlist up to rank 1.

Scope — read this before you use it

How to use

Rerank a first-stage retriever's shortlist. The model scores each pair; sort descending.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "quanfire-ai/rerank-statute-en"
tok = AutoTokenizer.from_pretrained(name)
ce = AutoModelForSequenceClassification.from_pretrained(name).eval()

query = "What is the presumptive taxation rate for eligible small businesses?"
candidates = [                      # e.g. the top-k a bi-encoder returned
    "Section 44AD. ... a sum equal to eight per cent of the total turnover ...",
    "Section 44AE. ... in respect of each goods carriage ...",
    "Section 80C. ... deduction in respect of life insurance premia ...",
]

with torch.no_grad():
    enc = tok([query] * len(candidates), candidates, padding=True,
              truncation=True, max_length=256, return_tensors="pt")
    scores = ce(**enc).logits.squeeze(-1)          # higher = more relevant

ranked = [c for _, c in sorted(zip(scores.tolist(), candidates), reverse=True)]

It is also loadable as a sentence_transformers.CrossEncoder(name) if you prefer that API.

Results

Evaluated the production way: a bi-encoder (multilingual-e5-small) retrieves the top-100 for each query, this cross-encoder rescores those 100, and we compare Recall@1. 95 % CIs are by paired bootstrap (B=2000) — both arms see the same queries, so the delta is bootstrapped paired and its interval excluding 0 is the significance test.

The split is document-isolated. Held-out queries are excluded by Act, not by query string: an Act is either wholly in training or wholly in evaluation, never split across both. This is the thing the withdrawn run got wrong, and it is why these numbers are lower.

• 1,494 held-out queries · pool of 1,205 unique statutory passages
• Training volume 26,252 rows (2.93× the 8,966-row first clean re-measure)
• Checkpoint chosen by held-out loss, not last step

• Delta = +0.0388 (+53.7 %), paired 95 % CI [+0.0241, +0.0535] — excludes 0, a statistically significant improvement.
• This is one draw of four. The other three returned +36.1 %, +55.6 % and +76.9 % on this same evaluation; the full table is at the top of this card. Quote the range, not this row.
• Recall@100 ceiling = 0.7557 — the fraction of queries whose gold passage the first-stage retriever surfaces at all. Everything above that is out of the reranker's reach.
• Of the recoverable queries (gold in the top-100), the reranker puts it at #1 for 14.1 %.

Read this before comparing to anything. Both caveats in the notice at the top of this card apply here: the delta is the joint effect of more data and best-checkpoint selection (not separable — no ablation was run), and this run used 77.8 % of the withdrawn run's volume, so it is not volume-matched against any earlier figure. The withdrawn +63.8 % is kept in the collapsed record above for provenance only and must not be quoted, differenced, or described as having been "corrected" to this.

How it was built (honest — the negatives lesson)

The first training run collapsed to random at evaluation. It trained on mined hard negatives (schedule stubs, repealed-section markers, OCR-garbled fragments) which are form-separable from clean section bodies — so the cross-encoder learned a query-independent shortcut ("does this look like a clean positive?") that crushed the training loss and carried zero discriminative power at eval, where every candidate in the retriever's shortlist is already a clean body.

The fix (this release) draws form-matched negatives: each training negative is another record's real positive — a clean statutory body, form-identical to the answer. Now the only way to pick the right passage among several clean passages is to read the query and judge relevance, so the training distribution matches the evaluation distribution and no shortcut exists. The general lesson: a reranker's negatives must match the candidate distribution it will see at inference, or it optimises a proxy.

The second lesson — how the first clean re-measure came back null. After the split defect was found, the re-measure ran on a document-isolated split and produced nothing: a delta whose interval spanned zero. That null was real, but it was measured through two suppressors of its own. A vestigial filter left over from the abandoned mined-negatives recipe was silently dropping 68.9 % of the training pairs — 8,152 rows survived where 26,252 were available — and the checkpoint scored was the last step rather than the best one. Removing the filter and scoring the best checkpoint is the whole difference between that null and the seeded draws at the top of this card — which is exactly why the two changes cannot be attributed apart, and why this card says so instead of picking one. The general lesson: a null is a claim too, and deserves the same audit you would give a positive result before you publish it.

Licence & provenance

• Base: intfloat/multilingual-e5-small (MIT).
• Training data: English central bare-Act text — 858 Central Acts sourced from a public Zenodo dataset (record 5088102, CC-BY-4.0) — mined into section-level query/passage pairs, with form-matched negatives drawn from the same pool. Under Indian Copyright Act §52(1)(q)(ii), the text of a Central Act (bare Act, no third-party headnotes or annotations) is not an infringement to reproduce; only bare statutory text is used.
• Weights: Apache-2.0. The reranker is non-reconstructive — it emits a relevance score and never reproduces the source text — so it is train-safe on §52-clean statutory text and the weights are cleanly licensable. The corpus itself is not redistributed.

Intended use & limits

Use it as the second stage behind a statute retriever (ideally embed-statute-en) to re-rank English central-statutory shortlists. Do not use it as a retriever, on court judgments (that is embed-legal-en), on state/subordinate legislation, or on non-English text — none of those are validated.

What the number does and does not license you to assume. Every figure on this card is measured on English central bare-Act text, with queries held out by whole Act, against a 1,205-passage pool. They are relative gains over a weak first stage — this checkpoint's absolute Recall@1 is 0.1111, and the retriever's own ceiling is 0.7557, so most of the headroom is upstream in retrieval, not here. Transfer to other corpora is not measured: this model showed nothing about judgments, state legislation, or other languages, and a sibling Quanfire retriever has already demonstrated that in-domain legal gains can go flat out-of-origin. Measure it on your own data before depending on it.