FinModernBERT-embed-large-v1

FinModernBERT-embed-large-v1

A 395M-parameter finance text-embedding model that beats the 7B Fin-E5 on FinMTEB Summarization (+0.109) and STS (+0.010) at 1/18th the size — built with ≈120 GPU-hours (~5 days) on a single 2× RTX 3090 workstation, through a fully decontaminated four-step pipeline: measured baseline → 5.65B-token domain MLM (DAPT, ~85 of those GPU-hours) → 270k synthetic contrastive pairs → multi-task InfoNCE (~8 h) → WiSE-FT weight interpolation (0 training).

Now on the official FinMTEB leaderboard (18 EN models): STS #2 (the top embedding model — only a lexical BOW baseline scores higher) and Summarization #2 (behind only voyage-3-large), with the single best FINDsum score on the board. See §2 for the full placement.

This card documents the entire journey: the goal, every training stage, every measured number, the failure that almost sank the release, and the trick that fixed it.

1. The goal

Build the strongest defensible finance embedder on a single encoder of ModernBERT-large size, evaluated on the full English FinMTEB benchmark (35 tasks, 7 task types), against the published state of the art — Fin-E5 (FinMTEB paper), a finance-adapted e5-mistral-7B. Target: win STS and Summarization outright, make Retrieval competitive.

"Defensible" means: every training stage runs behind a hard decontamination gate against all FinMTEB English eval sets (word-shingle overlap index; eval-source datasets and contaminated lineages like FiQA/ConvFinQA excluded outright), and the eval harness is the official FinMTEB fork end to end.

2. Headline results — full FinMTEB-EN, 35 tasks

Fin-E5 / e5-mistral rows are the published Table-1 numbers (weights are not public); this model was scored locally on the same FinMTEB fork with identical task mains (nDCG@10 retrieval, Spearman STS/Summ, MAP reranking, accuracy/v-measure/AP elsewhere).

Official FinMTEB leaderboard placement

The model is now listed on the official FinMTEB leaderboard (English board: 18 models, including voyage-3-large, OpenAI text-embedding-3-large/small, NV-Embed v2, e5-mistral-7B, gte-Qwen1.5-7B, bge-en-icl and Fin-E5). Position as of 2026-07-19, ranks computed from the leaderboard's own per-task scores by averaging tasks within each type:

Only four models on the board achieve two or more top-2 task-type finishes: voyage-3-large and text-embedding-3-large (closed-source APIs), Fin-E5 (7B, weights not public) — and this 395M open-weights model. The leaderboard's numbers for this model match the locally-measured scores reported below exactly.

3. The journey, stage by stage

Stage 0 — Measure the real gap (2026-06-29)

Everything starts from a measured baseline of the untrained base model against the finance SOTA, on a 15-task FinMTEB subset (2 STS + 3 Summarization + 10 Retrieval):

Two findings shaped the whole project: (a) untrained ModernBERT-large already matches Fin-E5 on STS — that lead must survive training; (b) Retrieval (0.051) and Summarization (0.118) are where the crown is won or lost.

Stage 1 — Domain-adaptive MLM pretraining, 5.65B tokens

Released separately as FinModernBERT-large-DAPT.

• Corpus: 185 GB raw finance pool (SEC filings, financial news, finance web) → cleaning + decontamination gate → 5,520,194 packed windows × 1,024 tokens ≈ 5.65B tokens.
• Run: 1 epoch, 43,127 steps (≈131k tokens/step), mask rate 0.30, peak LR 5e-5 cosine, bf16, full finetune, 2× RTX 3090 DDP. MLM loss 1.134 → 0.888.
• Measured honestly: DAPT alone barely moves zero-shot embedding scores (STS 0.446 / Summ 0.120 / Retr 0.056) — an MLM objective doesn't reshape sentence geometry. Its value appears downstream, and later it becomes the WiSE-FT anchor.

Interlude — the long-document discovery (+0.185 Summarization for free)

FinMTEB Summarization documents are huge (FNS averages ~290k chars ≈ 70k+ tokens). A zero-training experiment showed head-truncation was destroying the signal: chunk the document into ≤16 windows of ~506 body tokens, embed each, L2-normalize, mean-pool, re-normalize → Summarization 0.120 → 0.305 with no training at all (FNS alone: 0.223 → 0.707). This became the official doc-side embedding strategy.

Stage 2 — 270k synthetic contrastive pairs

• Teacher: an open-weights 35B-A3B MoE instruction model (AWQ, vLLM, tensor-parallel over both GPUs) generating from SEC full-text chunks.
• Generated: 100k+ query→passage and graded-STS pairs; 16,696 document↔summary positives + 16,700 mismatched (score-0) summary negatives — built because FinMTEB Summarization requires exactly this separation, and Fin-E5 never trained doc↔summary pairs at all (its own finance adaptation regressed Summarization 0.528 → 0.480); 1,452 term→definition pairs (early-stopped when the distinct-term pool saturated).
• Filtering: faithfulness checks, dedup, length gates, contamination gate (dropped: 2,628 dup / 308 length / 81 contaminated / 9 unfaithful) → 115,238 finance pairs.
• Blend with open general data (Investopedia 45k, HotpotQA 30k, NQ 30k, GooAQ 20k, AllNLI 30k) → 270,238 pairs, 59% finance: query_passage 198,870 · sts 36,520 · doc_summary 16,696 · doc_summary_neg 16,700 · term_definition 1,452.

Stage 3 — Contrastive training

• Hard-negative mining (positive-aware): top-4 negatives per finance anchor mined with intfloat/e5-large-v2, exact top-50 then filtered by sim(cand) ≤ 0.95 · sim(positive) to avoid false negatives; ≥95% of anchors got 4 negatives.
• Trainer: SentenceTransformer (DAPT encoder + mean pooling, max_seq 512). Multi-dataset training: CachedMultipleNegativesRankingLoss (InfoNCE, scale 20, cross-device in-batch negatives — 255 negatives/device) for retrieval-type data, CoSENT for graded STS pairs. Datasets: qp_finance 108,870 (4 mined negs each) · qp_general 90,000 · doc_summary 16,696 triplets · sts 36,520 · term_definition 1,446.
• Run: 2 epochs, 1,976 steps, effective 256 anchors/step, lr 2e-5 cosine, bf16, gradient checkpointing, 2× RTX 3090 DDP, ~8h wall. Epoch-2 mean loss 2.18 < epoch-1 2.77. Fully checkpointed/resumable (the run survived two deliberate pauses).

The crisis — and the WiSE-FT fix

Pure contrastive training delivered Retrieval 0.569 and Summarization 0.625 — but STS collapsed 0.450 → 0.371, below the pre-registered release gate (≥ 0.434 = Fin-E5 parity). Both epoch checkpoints failed it.

Instead of retraining, the zero-cost fix: interpolate encoder weights back toward the DAPT initialization (WiSE-FT): w = α·contrastive + (1−α)·DAPT, swept on the eval:

Two lessons: α=0.5 scores above both endpoints on STS (the classic WiSE-FT bump), and retrieval skill decays steeply toward the DAPT end — α=0.65 is the balance point that passes the gate while keeping the Summarization crown and most of the retrieval gain.

4. All 35 task scores

5. The journey in one table

6. Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BatuhanECB/FinModernBERT-embed-large-v1")

# Asymmetric retrieval: prefix queries and passages
queries = ["query: What drove the increase in operating expenses?"]
passages = [
    "passage: Operating expenses rose 12% year-over-year, driven primarily by "
    "increased headcount in R&D and higher cloud infrastructure costs.",
]
q = model.encode(queries, normalize_embeddings=True)
p = model.encode(passages, normalize_embeddings=True)
print(q @ p.T)

# Symmetric use (STS / clustering): prefix both sides with "passage: "
sents = ["passage: Net revenue increased 8%.", "passage: Sales grew by eight percent."]
emb = model.encode(sents, normalize_embeddings=True)

Prefixes matter — training baked literal query: / passage: prefixes in. Long documents (>512 tokens): reproduce the eval numbers by chunking into ≤16 windows of ~506 body tokens, embedding each, L2-normalizing, mean-pooling, re-normalizing.

Embedding dim 1024 · mean pooling · cosine similarity · max_seq 512 (native ModernBERT window is 8,192; 512 is the training configuration).

7. Limitations (measured, not hypothetical)

• Table/numerical-QA retrieval is the weakest axis (FinQA 0.163, TATQA 0.189): no table-derived pairs in this release's training mix.
• Scientific-topic clustering (FinanceArxiv S2S/P2P ≈ 0.21): scientific abstracts absent from the mix. General clustering is fine (Pii 0.86, MInDS 0.83).
• Headline pair-classification (≈0.62): finance-NLI-style pairs absent from the mix.
• Ectsum (0.167) stays hard for all dense models in this size class.
• English-only, finance-tuned; general-domain quality not guaranteed.

Planned next iteration: round-2 self-mining with this model, STS-preserving retrain (dropping the need for interpolation), synthetic table-QA pairs, finance-NLI headline pairs, and label-based contrastive for classification.

8. Licensing & data

Model weights: Apache-2.0. The DAPT corpus and synthetic pair set are not released (source licensing does not permit redistribution); the encoder does not memorize/regurgitate corpus text in embedding use. Every training stage was decontaminated against all FinMTEB English eval sets before training.

Related

• FinModernBERT-pairs-sec-synthetic-v1 — the 115,238 SEC-synthetic training pairs (finance portion of the Stage-3 mix), released with the full blend recipe.
• FinModernBERT-large-DAPT — the Stage-1 finance-MLM foundation (and WiSE-FT anchor) for this model.
• ModernBERT-large — base model.
• FinMTEB — benchmark (paper).