l24-pls8-support-reranker-v1

L24 PLS8 Support Reranker v1

l24-pls8-support-reranker-v1 is an experimental second-stage evidence reranker. It operationalizes a compact query-conditioned support subspace from Qwen3-4B layer-24 prefill activations. It does not accept raw text by itself and does not generate answers.

Research Materials

For a structured review, start with:

• Reviewer guide
• Consolidated methods
• Application research summary
• Final claim-remediation report
• Novelty audit
• Machine-readable claim ledger
• Full source and tests

The Hugging Face repository mirrors the authoritative methods, final reports, and compact audit summaries. GitHub remains canonical for executable source, tests, the complete Spec Kit history, and the chronological report archive.

The model files and reported model-specific metrics on this card are unchanged. A July 2026 claim-remediation pass added stronger mechanistic controls and a separate L24+L34 research finalist. That finalist is not part of this model repository. The remediation narrows the general claim: activation telemetry reliably improves over dense retrieval in the evaluated settings, but does not generally outperform Ettin.

Model Summary

For every candidate, the telemetry backbone captures two 2,560-dimensional layer-24 resid_pre decision states:

• the query plus candidate under an evidence-support prompt;
• the same candidate under a candidate-only support-prior prompt.

The released projection computes:

delta = h(query, candidate) - h(candidate only)
clean_delta = delta - (delta @ U_nuisance) @ U_nuisance.T
z8 = ((clean_delta - support_mean) / support_scale) @ PLS_rotations

The rank-9 nuisance basis removes candidate-intrinsic and shuffled-query-persistent structure. The eight PLS coordinates are concatenated with dense score, within-slate dense z-score, and reciprocal dense rank.

Architecture:

Linear(11, 128)
ReLU
Linear(128, 128)
ReLU
Linear(128, 1)

The final ranking score is:

0.2 * z(dense_score) + 0.8 * z(MLP_score)

The artifact uses Safetensors and NumPy arrays; it contains no executable pickle.

Intended Use

Use this artifact for research or high-value second-stage reranking over an organic dense top-100 slate, especially where near-topic distractors are costly:

• legal evidence and contract-clause retrieval;
• biomedical or clinical evidence retrieval with expert oversight;
• scientific, policy, compliance, and technical-documentation retrieval;
• mechanistic-interpretability research on behaviorally relevant activation subspaces.

It is not a first-stage retriever, text-only reranker, generator, safety classifier, or autonomous legal/medical decision system.

Installation

pip install 'activation-rag @ git+https://github.com/araujota/activation-rag.git'
hf download araujota/l24-pls8-support-reranker-v1 \
  --local-dir l24-pls8-support-reranker-v1

Usage

Each JSONL row contains one query slate. Candidates require chunk_id, dense_score, dense_rank, and either eight precomputed coordinates:

{"query_id":"q1","candidates":[
  {"chunk_id":"c1","dense_score":0.81,"dense_rank":1,
   "l24_support_coordinates":[0.1,-0.2,0.3,0.0,0.2,-0.1,0.4,0.1]}
]}

or paired raw layer-24 states:

{"chunk_id":"c1","dense_score":0.81,"dense_rank":1,
 "l24_true_pair_resid_pre":["2560 floats"],
 "l24_candidate_only_resid_pre":["2560 floats"]}

activation-rag-score \
  --bundle l24-pls8-support-reranker-v1 \
  --input slates.jsonl \
  --output rankings.jsonl

Python:

from activation_rag.release_reranker import L24PLS8SupportReranker

reranker = L24PLS8SupportReranker.load("l24-pls8-support-reranker-v1")
coordinates = reranker.project_support_coordinates(true_state, candidate_state)
ranking = reranker.score_slate(candidates)

The complete capture commands and production integration notes are in the GitHub README.

Prompt And Capture Contract

True pair:

Query:
{query}

Candidate evidence:
{candidate}

Task:
Does the candidate evidence directly support answering the query? Focus on exact support, not topic similarity.

Answer support:

Candidate-only prior:

Candidate evidence:
{candidate}

Task:
Assess whether this evidence looks answer-supporting.

Support signal:

Capture must use the released Qwen3-4B/RMT telemetry backbone, the final prompt position, layer 24 resid_pre, zero generated tokens, and an early stop after layer 24. Prompt, tokenizer, checkpoint, adapter, site, and decision position are part of the representation contract.

The backbone is documented at araujota/activation-rag-telemetry-backbone-v1.

Training

• Training domains: LegalBench-RAG and pooled R2MED.
• Candidate slates: organic dense top-100 only; no qrel-positive insertion.
• Nuisance fit: query-grouped train-only cross-validation.
• Selected nuisance ranks: candidate prior 8, shuffled-query persistence 1.
• PLS fit: 37,906 train rows, including 5,978 positive rows and top-eight dense hard negatives per positive-containing query.
• MLP loss: query-listwise softmax ranking objective.
• Hidden width: 128.
• Checkpoint and alpha selection: combined LegalBench-RAG/R2MED dev only.
• Selected artifact: seed 13, epoch 5, alpha 0.8.
• APPS: excluded from all fitting and selection and opened only after artifact freeze.

Evaluation

All values use full query denominators over organically retrieved dense top-100 candidates.

Combined PLS8+dense versus Ettin:

• nDCG delta +0.0472, 95% CI [+0.0242,+0.0703], p=0.0001;
• MRR delta +0.0527, 95% CI [+0.0267,+0.0788], p=0.0001;
• Recall delta +0.0418, 95% CI [+0.0161,+0.0679], p=0.0024.

The nDCG gain over Ettin is independently significant on LegalBench-RAG (+0.0399, p=0.0018) and R2MED (+0.0857, p=0.0025).

Three seeds reach combined nDCG 0.4081, 0.4067, and 0.4029. After removing 118 queries with exact query-text or positive-evidence overlap, PLS8+dense still gains +0.1312 nDCG over dense on 702 queries.

APPS Zero-Shot

The gain over dense is significant (+0.0243 nDCG, 95% CI [+0.0127,+0.0363], p=0.0001), but the artifact remains below Ettin. This is evidence of positive transfer, not universal superiority.

Interpretability Basis

The rank-8 support subspace was promoted after decomposition and causal analysis showed that:

• raw layer-24 interaction residuals carry a coordinated support-like signal;
• the validated fixed-TopK SAE loses much of it into reconstruction error;
• the cleaned component remains active after candidate-prior nuisance removal;
• norm-matched activation patching changes downstream support-decision logits in the predicted direction.

The eight PLS axes are compact distributed coordinates, not eight individually human-labeled concepts. Later exact controls support a broader, causally active late-layer evidence-judgment process at L22/L24, but reject stronger claims of a monosemantic relevance variable, fully localized circuit, or independently causal L34 feature. Nuisance-erased coordinates remain predictively useful, but their intervention effect does not beat candidate-reassigned controls.

See the claim-remediation report and claim ledger.

Limitations And Risks

• This is activation-augmented reranking, not activation-only retrieval. Dense metadata enters the MLP and final blend.
• It cannot recover relevant evidence absent from the initial candidate slate.
• The strongest result uses two trained expert-prose domains. Independent cross-domain replication is required.
• This artifact's strong LegalBench-RAG/R2MED results do not establish general superiority over text rerankers; the broader corrected conclusion is domain- and system-dependent.
• Scores are slate-relative because both dense and MLP outputs are z-normalized.
• Two early-stopped prefills per candidate are expensive. Batch, cache the candidate-only posture, truncate after layer 24, and gate usage in production.
• Exact telemetry dependencies are required; changing the representation contract invalidates calibration.
• Benchmark relevance labels do not establish factual, legal, or clinical correctness.

The previous activation-support-fusion-v1 remains available for reproducibility but is superseded. The historical behavior-latent-general-v0.1 result remains withdrawn because of contaminated training or qrel-positive diagnostic slates.

Reproducibility

The bundle includes safe weights, frozen projection arrays, normalizers, exact feature order, fit/dev/test query manifests, evaluation summaries, and SHA-256 checksums. Run:

python scripts/verify_l24_pls8_release.py l24-pls8-support-reranker-v1

The complete report is docs/l24-pls8-reranker-report-20260717.md.

The later publication-grade due-diligence report is docs/fellowship-claim-remediation-report-20260723.md.