Downloads · 30 days
0
enjalot/sae-jina-colbert-v2-tokens-64K
sae-jina-colbert-v2-tokens-64K is a machine learning model from enjalot. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
A 65,536-feature BatchTopK sparse autoencoder on per-token late-interaction embeddings from jinaai/jina-colbert-v2 (128-d token vectors). Unlike our pooled sentence-embedding SAEs, this decomposes every token vector o…
Downloads · 30 days
0
Access
Public
Updated Jul 4, 2026
Repo size
67.4 MB
Likes
0
Public
Click a slice to open those files.
.safetensors67.4 MB · 100%
From the Hugging Face model README
A 65,536-feature BatchTopK sparse autoencoder on per-token late-interaction embeddings from
jinaai/jina-colbert-v2 (128-d token vectors).
Unlike our pooled sentence-embedding SAEs, this decomposes every token vector of a ColBERT
document into sparse features — to our knowledge the first late-interaction (ColBERT) SAE whose
features are interpretable at population scale (all 65,487 live features labeled and judged, 58.9%
coherent+thematic, vs 3.6% for the prior mxbai-edge 64-d token SAE under the same judging pipeline).
This card follows a standing house rule: every quantitative claim ships with its baseline, its confidence interval, and the number of seeds behind it. Where an artifact has no CI, we say so instead of inventing one.
| SAE type | BatchTopK (batch-level top-k during training; learned EMA activation threshold at inference, capped at k) |
| Input dim | 128 (one jina-colbert-v2 token vector) |
| Latents | 65,536 (512× expansion) |
| k (train) | 32 per token; inference L0 is threshold-gated, mean ≈ 25.5 |
| Training | batch 8192, LR warmup 2,000 steps, AMP, unit-norm decoder, grad clip, auxk off |
| Seeds | 1 training seed (data seed 42) — see seed-stability caveat below |
| Checkpoint | sae_batchtopk_32_512.pooled (final), subfolder 32_512/ |
Per-token document embeddings of the English FineWeb-Edu 10BT sample (500-token chunks), embedded with jina-colbert-v2: 506,191,906 unique token vectors, replayed 4× → 2,024,767,488 training samples (2.02B tokens). Trained in 89 min on one RTX 5090 at ~377K tokens/s. The disk-backed replay recipe matters: an on-the-fly variant of this pipeline measured lower feature coherence at equal token count (43.9% vs 54.8% token-rendered COH+THM, CIs separated).
The SAE consumes document-side token embeddings produced exactly as recorded in the training
corpus' embedding_meta.json:
jinaai/jina-colbert-v2 @ revision 4552c4dc (trust_remote_code=True), via
pylate.is_query=False; queries use a different marker and
were not part of training data).Feed the SAE individual token vectors (any batch of [n_tokens, 128]); it has no notion of
sequence.
Held-out eval (never-trained 259K-chunk token set, n=200K token rows, 1 seed):
| metric | value | context |
|---|---|---|
| FVU | 0.0767 | train FVU ≈ held-out −0.001 → no generalization gap; 32K sibling: 0.0825 |
| L0 | 25.5 | threshold-gated (k=32 cap) |
| Dead features | 0.20% (131 of 65,536) | at 512× overcompleteness; vanilla TopK at this scale ran ~15% dead |
| MMCS | 0.408 |
For calibration: the internal ship-gate FVU bar of ≤0.07 was calibrated on a 64-d token space; this 128-d model sits marginally above it (0.0767) while passing the two function-tracking criteria (dead%, retrieval recovery) strongly.
SciFact, late-interaction MaxSim scoring, n=300 queries, 1 seed (point estimates, no CI in the artifact):
| system | nDCG@10 | recovered | rank corr vs raw |
|---|---|---|---|
| raw jina-colbert-v2 | 0.6938 | — | 1.0 |
| this SAE (64K), reconstruction | 0.6796 | 98.0% | Spearman 0.907 |
| 32K sibling, reconstruction | 0.6637 | 95.7% | Spearman 0.917 |
All 65,487 live features (99.93% of the dictionary) were rendered as span exemplars (±16 tokens around the firing token, fired tokens highlighted, 10 windows/feature, stratified extraction) and labeled + judged with qwen2.5:32b:
| class | share of 65,487 |
|---|---|
| COHERENT | 27.3% |
| THEMATIC | 31.6% |
| COH+THM | 58.9% (38,602 features) |
| GENERIC | 33.0% |
| POLYSEMANTIC | 8.0% |
| UNCLEAR | 0.06% |
_fixed) label shards.Granularity ladder (same 250 random live features, seed 42, rendered three ways, Wilson 95% CIs — how coherence depends on how much context you show the judge):
| rendering | COH+THM [95% CI] |
|---|---|
| token window (±10) | 54.8% [48.6, 60.9] |
| span ±16 + highlighted fired tokens | 63.6% [57.5, 69.3] |
| full chunk + highlighted fired tokens | 68.0% [62.0, 73.5] |
Fired-token highlighting is the mechanism that matters (POLYSEMANTIC 18.8% → 4.4%); chunk rendering wins COH+THM but by THEMATIC inflation (COHERENT actually drops to 22.4%); span is the sweet spot (best COHERENT cell, 32.4%).
Two findings from the same research program motivated this model:
Hence: strongest available late-interaction encoder (559M-param jina-colbert-v2, 128-d tokens) + the BatchTopK ≥2B-token replay recipe.
Reconstruction and labeling tell you the features describe the token space; these interventions show they drive late-interaction retrieval. Protocol (identical to the mxbai ColBERT V4 runs): per query, take the top-100 MaxSim pool; a feature's distinctiveness = its summed MaxSim attribution on relevant pool docs minus non-relevant; ablate the top-m distinctive features in recon space (subtract only those features' contribution to each token — proportional, not whole-token masking) and rerank. The control ablates m frequency-matched random features (matched on corpus firing count — mandatory: distinctive features also fire often, so an unmatched random baseline fakes a large effect). ΔnDCG@10 with percentile-bootstrap 95% CIs (n=2000).
Necessity — recon space, dose-response (SciFact n=282 queries, TREC-COVID n=50, 1 seed):
| dataset | m | ΔnDCG distinctive [95% CI] | ΔnDCG freq-matched random [95% CI] |
|---|---|---|---|
| SciFact | 5 | −0.123 [−0.149, −0.099] | +0.000 [−0.000, +0.001] |
| SciFact | 10 | −0.204 [−0.237, −0.172] | +0.004 [+0.001, +0.008] |
| SciFact | 25 | −0.323 [−0.365, −0.286] | +0.006 [+0.001, +0.011] |
| SciFact | 50 | −0.424 [−0.469, −0.383] | +0.011 [+0.004, +0.018] |
| TREC-COVID | 5 | −0.148 [−0.206, −0.096] | +0.004 [−0.002, +0.012] |
| TREC-COVID | 10 | −0.194 [−0.252, −0.139] | −0.001 [−0.015, +0.013] |
| TREC-COVID | 25 | −0.280 [−0.345, −0.217] | −0.004 [−0.020, +0.012] |
| TREC-COVID | 50 | −0.353 [−0.421, −0.285] | −0.008 [−0.024, +0.008] |
Ablating relevance-distinctive features drops MaxSim nDCG monotonically with dose, on both datasets, while frequency-matched random features barely move it (random CIs mostly straddle zero). At the reference dose m=25 the necessity gap is +0.329 (SciFact) / +0.276 (TREC-COVID) — roughly 3.5× the earlier mxbai-edge 64-d ColBERT SAE's +0.081 at the same dose, consistent with this model's much higher feature coherence. (Coarse match-mask cross-check at m=25: distinctive −0.604 / −0.524 vs random −0.024 / −0.106 — same ordering, larger magnitudes.)
Steering — controllability (SciFact, 40 features each firing on ≥50 corpus tokens): append one synthetic query token = α·(unit decoder direction of feature f); measure the change in the number of f-firing tokens among the top-10 retrieved docs, vs a norm-matched random direction.
| α | steer toward f [95% CI] | random direction [95% CI] |
|---|---|---|
| 2 | +0.41 [+0.32, +0.51] | −0.02 [−0.03, +0.00] |
| 4 | +1.58 [+1.07, +2.21] | −0.00 [−0.02, +0.02] |
| 8 | +5.98 [+4.27, +7.91] | +0.04 [−0.02, +0.11] |
Pushing a query toward a feature monotonically surfaces documents that fire it (clean dose-response, CIs separated from the random control at every α); a random direction does not. Combined with necessity, the features are bidirectional causal handles on late-interaction retrieval — remove them and relevant docs fall, steer toward them and matching docs rise.
Scope: faithfulness recovered = 97.5% (SciFact) / 88.5% (TREC-COVID) under this intervention's
encode/decode path (SciFact matches the 98.0% headline from the retrieval eval's forward path).
1 training seed; per-feature steering lifts are high-variance (features differ widely in how many
docs they fire on) but the mean is well above the random control throughout. Full numbers:
intervention_colbert_{scifact,trec-covid}.json, intervention_colbert_scifact_m{5,10,50}.json,
steering_colbert_scifact.json in the training run dir.
Across retrainings with different seeds, only ~4.6% of ColBERT-token SAE features reproduce as near-twins (decoder cosine ≥0.7; exact 1-1 bipartite matching 4.5–4.6%, measured on the 96-d ceiling models, 2-seed pairs; not yet measured for this exact run). This model is 1 training seed: trust aggregates and validated features, not individual feature indices.
The 2026-07-01 BEIR shingle scan fully covers this model's training data: it scanned all 11 on-disk fineweb-edu chunked-500 parquets in full (2,890,362 chunks, per-file row counts verified), and this SAE trained on the first 2,000,000 of those chunks (506M tokens) — a strict subset of the scanned set. Against that superset, SciFact had 16 docs with ≥2 shingle hits (0.31% of 5,183) and TREC-COVID 147 (0.10% of 145,043 shingled); exclusion-rescoring the contaminated docs moved pooled-SAE nDCG@10 by <0.005 in all conditions (metric-neutral).
Requires latentsae >= 0.2.0 (BatchTopK support; not on PyPI, which is still 0.1.3 —
install from GitHub) and pylate:
# pip install git+https://github.com/enjalot/latent-sae pylate
import torch
from latentsae import Sae
from pylate import models
sae = Sae.load_from_hub("enjalot/sae-jina-colbert-v2-tokens-64K", "32_512")
model = models.ColBERT(
model_name_or_path="jinaai/jina-colbert-v2", trust_remote_code=True
)
# DOCUMENT-side token embeddings (the space the SAE was trained on)
doc_tokens = model.encode(
["Sparse autoencoders decompose embeddings into interpretable features."],
is_query=False,
)
x = torch.tensor(doc_tokens[0], dtype=torch.float32) # [n_tokens, 128]
features = sae.encode(x) # per-token sparse codes
print(features.top_indices) # which of the 65,536 features fired per token
print(features.top_acts) # how strongly (threshold-gated, <= 32 active)
recon = sae.decode(features.top_acts, features.top_indices) # back to 128-d tokens
Trained 2026-07-02 with enjalot/latent-sae
(run colbert_jinacolbert128_batchtopk_64K_k32_2B); held-out metrics + SciFact MaxSim eval
2026-07-03, full-population span labeling 2026-07-03, granularity ladder 2026-07-03, CJK label
repair 2026-07-04, causal necessity + steering (SciFact + freshly-embedded full-corpus TREC-COVID)
2026-07-04. Sibling pooled-embedding releases:
enjalot/sae-jina-v5-nano-multilingual-98K,
enjalot/sae-all-MiniLM-L6-v2-stagej-49K.