sae-all-MiniLM-L6-v2-stagej-49K

sae-all-MiniLM-L6-v2-stagej-49K ("MiniLM Stage-J")

A 49,152-feature Matryoshka sparse autoencoder on pooled sentence-transformers/all-MiniLM-L6-v2 embeddings (384-d) — interpretability for the small, fast, extremely widely used workhorse embedding model. It mirrors the jina-v5-nano "Stage J" recipe (Matryoshka nesting, k=64, 128× overcompleteness) and is the successor to the older enjalot/sae-all-MiniLM-L6-v2-FineWeb-RedPajama-Pile-150M TopK SAEs (1.5K–3K latents): ~16–32× more features with far better utilization (2.9% dead).

This card follows a standing house rule: every quantitative claim ships with its baseline, its confidence interval, and the number of seeds behind it. Where an artifact has no CI, we say so instead of inventing one.

Architecture & config

Training data

English 3-corpus mix of ~120-token chunks embedded with all-MiniLM-L6-v2 (default encode() document embeddings, no prompts — MiniLM has no prompt convention):

• FineWeb-Edu (10BT sample), RedPajama-Data-V2 (10B sample), Pile-uncopyrighted

153,373,857 unique chunk embeddings (≈18.4B source tokens at ~120 tokens/chunk), replayed 2× → 306,747,392 training samples (~300K steps at batch 1024). Trained in 5.7 h on one RTX 5090.

Embedding contract

Feed the SAE plain model.encode(texts) all-MiniLM-L6-v2 embeddings — the same distribution it was trained on. There is no query/document prompt asymmetry for MiniLM (unlike our jina-v5-nano Stage J release, where queries take a "Query: " prefix). All evals below are self-consistent in this default embedding space.

Reconstruction quality

Held-out eval (200K-row tail of a FineWeb shard, 1 seed):

Full level: FVU 0.139, L0 = 64, 2.9% dead (47,703 of 49,152 features live). Monotonic per-level FVU confirms the Matryoshka nesting behaves correctly. Reconstruction matches jina Stage J (~0.13) at half the feature count on a 2× smaller embedding.

Interpretability — honest status

• All 45,059 features firing ≥10× were auto-labeled (qwen2.5:32b, top + decile-stratified exemplars + decoder-NN control samples; 45,059/45,059 succeeded): 79% distinct labels (vs 98% for the published nomic-24K SAE labeled with the same pipeline — a notch behind, as expected for a smaller 384-d embedding), ~30% generic-ish, median label length 4 words. ~26% share a generic bucket ("historical figures and events" ×281); the other 74% are distinct and mostly specific ("social security disability benefits", "docker containers and images"). 0.16% malformed labels (72 features, qwen drift). Labels are not bundled in this repo.
• Stratified LLM-judged coherence (the honest COH+THM protocol): 46.1% [41.8, 50.4] (n=512 random live features, decile-stratified exemplar windows, qwen2.5:32b judge, Wilson 95% CI; measured 2026-07-04). Class split: COHERENT 11.9 / THEMATIC 34.2 / GENERIC 4.7 / POLYSEMANTIC 49.2. This sits below the jina Stage J reference (60.7 [56.4, 64.9], CIs disjoint); the gap is polysemanticity (49.2% vs 23.4%), not generic noise — consistent with a smaller 384-d embedding packing more concepts per feature. The judge is uncalibrated against human raters (audit pending), the same caveat carried by all coherence numbers in this series. Top-N-style scoring inflates COH+THM by ~9–14 pts on our models; treat any top-N number you compute yourself as an upper bound.

Causal validation (retrieval, with controls and CIs)

All intervention CIs are 95% percentile bootstrap over queries; controls are m random live features and m frequency-matched random features. This checkpoint, 1 training seed. Faithfulness covers 2 datasets; the necessity/sufficiency/steering triad is SciFact-only for this model (it replicates the pattern established on jina Stage J across 2 datasets).

Faithfulness (does the SAE preserve retrieval?):

Point estimates (no CI in the artifact). TREC-COVID uses the 2026-07 corrected harness (exponential nDCG gain, full corpus — the pre-correction capped harness read 0.592 raw and is void); recon > raw is the known SAE-as-denoiser effect, also seen on jina Stage J.

Necessity (ablate the query's relevance-distinctive features; SciFact):

Sparse-space effect saturates to ≈0 at m=50 (ranked list runs past the truly distinctive features). "Full-level" = level-aware matryoshka ablation (zero the feature at every nested level).

Sufficiency (keep ONLY the top-m distinctive features of the query's 64-feature code; fraction of full-code pool nDCG recovered, pool_k=100, SciFact):

5–10 features recover ≈1.0–1.17× of full-code retrieval vs ≈0.06–0.10 controls (recovery >1 is oracle-selection denoising: masking distractor features helps).

Steering (push queries along decoder directions; 60 features, top-10 lift, SciFact):

Monotonic dose-response, all CIs exclude zero, random-direction controls ≈0.

Cross-model note: this model passing the full necessity + sufficiency + steering triad replicates the jina Stage J causal result on a second embedding model; the 3-seed necessity replication (−0.130 to −0.155, all CIs exclude 0) was run on a companion vanilla-TopK jina SAE, not on this checkpoint.

Stability caveat

Across retrainings with different seeds, only ~11% of features of a comparable dense SAE reproduce as near-twins (decoder cosine ≥0.7, exact 1-1 bipartite matching, 3 seeds of a companion jina 24K SAE; not yet measured for this MiniLM run). Trust aggregates and validated features, not individual feature indices.

Leakage note

A full shingle scan of the 162.5M-chunk training universe (including these exact chunked-120 corpora) against BEIR found real but metric-neutral contamination: verbatim full-abstract copies for 0.8–1.4% of SciFact docs and ~0.9% of TREC-COVID docs (Pile is the main source). Rescoring with contaminated docs excluded moves every nDCG@10 by < 0.005, so numbers are reported unadjusted with this caveat.

Usage

Requires the GitHub version of latentsae (Matryoshka support is not in the PyPI 0.1.3 release):

# pip install git+https://github.com/enjalot/latent-sae
from latentsae import Sae
from sentence_transformers import SentenceTransformer

sae = Sae.load_from_hub("enjalot/sae-all-MiniLM-L6-v2-stagej-49K", "64_128")

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(
    ["Hello world", "Sparse autoencoders decompose embeddings"],
    convert_to_tensor=True,
)

features = sae.encode(emb)
print(features.top_indices)  # which of the 49,152 features fired (64 per input)
print(features.top_acts)     # how strongly

recon = sae.decode(features.top_acts, features.top_indices)  # back to 384-d

Provenance

Trained 2026-06-20 with enjalot/latent-sae (run minilm_l6_stagej_49K_oldrecipe_replay2_k64); labeling 2026-06-21, causal triad 2026-07-01, TREC-COVID harness correction 2026-07-01. Sibling release: enjalot/sae-jina-v5-nano-multilingual-98K.