activation-support-fusion-v1

Activation Support Fusion v1

activation-support-fusion-v1 is an experimental second-stage evidence reranker. It combines prefill-only Qwen3-4B telemetry with metadata from an organic dense top-100 candidate slate. It does not accept text directly and it does not generate answers.

Model Details

• Architecture: Linear(254,128) -> ReLU -> Linear(128,128) -> ReLU -> Linear(128,1)
• Safe weights: model.safetensors
• Base model used for telemetry: Qwen/Qwen3-4B with the released RMT adapter
• Layer-7 input: 245 SAE magnitudes from a uniform 256-token capture window, max-pooled per feature and transformed with signed log1p plus row L2 normalization
• Dense inputs: raw score, within-slate z-score, reciprocal dense rank
• Layer-24 inputs: six raw/slate-relative/interaction derivatives of a decision-position resid_pre support residual
• Final score mixing alpha: 1.0; dense metadata remain MLP inputs
• Candidate contract: organic dense top-100, no qrel-positive insertion

The layer-24 primitive is computed from two zero-generation prefills:

row_l2(hidden_l24(true query + candidate) - hidden_l24(candidate only))

It is projected onto a split-safe L1-logistic support direction. The resulting scalar is expanded with within-slate z-score, rank, disagreement, and dense interaction terms before MLP inference.

Intended Use

Use this artifact for research on activation-augmented reranking over a frozen dense candidate slate, particularly expert evidence retrieval where topical hard negatives are common. Appropriate uses include controlled legal, biomedical, scientific, policy, and technical-documentation experiments.

Do not use it as a first-stage retriever, text-only reranker, answer generator, safety classifier, or autonomous medical/legal decision system.

Usage

pip install 'activation-rag @ git+https://github.com/araujota/activation-rag.git'
hf download araujota/activation-support-fusion-v1 --local-dir activation-support-fusion-v1
activation-rag-score \
  --bundle activation-support-fusion-v1 \
  --input slates.jsonl \
  --output rankings.jsonl

Each JSONL row has query_id and candidates. Each candidate requires:

• chunk_id, dense_score, and positive integer dense_rank;
• core245, either a feature-ID mapping or 245-value ordered array;
• l24_support_raw, or the two 2,560-value hidden-state arrays from which it is derived.

from activation_rag.release_reranker import ActivationSupportFusionReranker

model = ActivationSupportFusionReranker.load("activation-support-fusion-v1")
ranked = model.score_slate(candidates)

The complete prompt, telemetry-capture, and production integration instructions are in the GitHub README.

Evaluation

The corrected publication evaluation used held-out LegalBench-RAG and pooled R2MED queries, organic dense top-100 candidates, no qrel-positive appending, and every test query in the metric denominator.

Full fusion versus dense: absolute +0.0981, relative +36.5%, paired randomization p=0.00010, bootstrap 95% CI [0.0797, 0.1166], 820 queries. LegalBench-RAG was 0.3714 versus dense 0.2820; R2MED was 0.3404 versus dense 0.1964.

The full model is statistically tied with the strongest tested text rerankers. Its Core245 increment over L24+dense is modest but replicated (+0.0129, p=0.0196; GPU replay +0.0117, p=0.0399). The release does not claim universal state of the art.

Training And Data Controls

The support direction and MLP were fit only on train partitions. Model and hyperparameter selection used dev partitions. Test query IDs were held out and are included in this repository for audit. Candidate-group guards required:

• strict_organic_dense_topk_no_qrel_append provenance;
• at most 100 candidates per query;
• top-ranked selection rather than qrel-driven candidate inclusion;
• no train/dev/test query overlap;
• complete L24 score coverage before fusion materialization.

The MLP used groupwise listwise ranking, AdamW, dense hard negatives, dropout, weight decay, and a dev-selected alpha/epoch. See config.json, l24-training-summary.json, and evaluation-summary.json for the released contract and metrics.

Limitations And Risks

• This is not raw-text inference: exact telemetry dependencies are required.
• Results are strongest on two expert-prose datasets and need independent, cross-domain replication.
• Activation-only L24 retrieval underperformed dense; the successful system is fusion.
• Dense score distributions and L24 slate statistics make scores slate-dependent.
• Prompt, tokenizer, base checkpoint, adapter, layer, site, position, SAE, and pooling changes invalidate calibration.
• Two prefills per candidate are expensive. Cache candidate-only states, batch candidates, and stop the model after layer 24 in production.
• Benchmark relevance labels do not establish factual, legal, or clinical correctness.

Historical high scores from behavior-latent-general-v0.1, including an APPS diagnostic, were affected by contaminated training or qrel-positive candidate insertion and are explicitly withdrawn.

Reproducibility

release-manifest.json supplies SHA-256 hashes for every release file. The bundle uses Safetensors and a NumPy scorer, avoiding executable pickle loading. The exact feature order and normalizer are embedded in config.json and checked against feature-manifest.json at load time.

Citation metadata will be added after archival publication. For now, cite the repository URL and immutable Git commit used in your experiment.