ProtSent-V2.5-35M

ProtSent-V2.5 ESM-2 35M

ProtSent-V2 35M plus one more contrastive pass on a fresh draw of the corpus, with a DMS/ProteinGym CoSENT target and a Global Orthogonal Regularization term added.

Mean-pooled ESM-2 35M embeddings, dimension 480, Matryoshka heads at 64/128/256.

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("GrimSqueaker/ProtSent-V2.5-35M")
emb = model.encode(["MKTLLLTLVVVTIVCLDLGYT", "MKTLLLTLVVVTIVCLDLGYN", "AGWYRSPQEGLKPVDTFKDIV"])
print(cos_sim(emb[0], emb[1:]))

Training

Corpora are decontaminated with MMseqs2 at 40% identity / 80% coverage against the remote-homology and PPI test splits. The DMS parquet is not decontaminated — four suite tasks are DMS-derived (Stability, Fluorescence, beta-lactamase, Variant Effect), and only exact-match overlap has been checked (zero on all three tested).

Results

SCOPe-40 structural retrieval, test split, self excluded, restricted to the 1,693 of 2,207 queries that have a non-self same-family protein in the gallery.

Paired bootstrap over queries, 10,000 resamples, V2.5 − V2: R@1 +0.0077 [−0.0041, +0.0201], R@10 +0.0018 [−0.0053, +0.0089], R@30 +0.0047 [+0.0000, +0.0100], MAP +0.0078 [+0.0032, +0.0125]. MAP is the only metric excluding zero, so the supportable claim is ranking depth, not top-1. Profile alignment (HMMER) still leads at R@1.

23-task downstream suite. Win/tie/loss over the 20 tasks with a defined one-vs-rest AUC (ties = |delta| < 0.005), with the median delta:

V2.5 is indistinguishable from V2 on this aggregate. A sign test resolves almost none of these records, so no inferential claim is drawn from the tallies. The linear-probe deficit against vanilla ESM-2 is unchanged.

Largest per-task moves, V2.5 − V2, linear probe: Stability +0.0946, AAV Fitness +0.0732, Fluorescence +0.0149, Variant Effect +0.0128, beta-lactamase +0.0110, Optimal pH −0.0247, Binary Subcellular Localization −0.0128. The gains are the DMS-derived tasks, matching the re-added CoSENT target.

An ablation trained with --gor_weight 0 and everything else identical matches this model within noise (2W/15T/2L on the k-NN suite; SCOPe-40 eligible R@1 0.6923 / MAP 0.6528). GOR cost +11.7% per step and is not the source of the improvement over V2.

Training code and full run log: github.com/oriel9p/ProtSent (train_esm2_35m_v2p5.sh, RUNS.md).