ettin-encoder-32m-pretrain-50m

ettin-encoder-32m-pretrain-50m

A retrieval encoder pretrained from jhu-clsp/ettin-encoder-32m on a 50M-pair balanced split of the DenseOn embeddings pre-training dataset.

Training data: this model was contrastively pretrained on capemox/denseon-pretrain-50m-balanced, a 50,000,000-pair sample drawn from the DenseOn corpus lightonai/embeddings-pre-training-curated (665M curated query–document pairs across 34 sources). Pairs are sampled with T=2 temperature weighting + iterative equal-redistribution capping so that no single source dominates the mix.

This is a Stage 1 (pretraining-only) checkpoint — it has not been fine-tuned on supervised retrieval data. Use it as a strong starting point for Stage 2 fine-tuning, or as a zero-shot retrieval encoder.

Training recipe

Evaluation (Stage 1 checkpoint, NDCG@10)

BEIR subset (zero-shot, no fine-tuning):

NanoBEIR (13-dataset aggregate, end of training): 0.5300

Downstream value

When this checkpoint is fine-tuned on MS MARCO hard negatives (tomaarsen/msmarco-Qwen3-Reranker-0.6B), it reaches 0.3225 mean BEIR NDCG@10 — versus 0.2319 for the same fine-tune starting from the raw base model (+39% from this pretraining step).

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("capemox/ettin-encoder-32m-pretrain-50m")

queries = ["What is the capital of France?"]
docs    = ["Paris is the capital and largest city of France."]

q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs,    normalize_embeddings=True)
print(q @ d.T)