FinModernBERT-large-DAPT

FinModernBERT-large-DAPT

ModernBERT-large domain-adapted to finance with 5.65B tokens of masked-language-model pretraining. This is the Stage-1 foundation of a pipeline whose final embedding model, FinModernBERT-embed-large-v1, beats the 7B Fin-E5 on FinMTEB Summarization (+0.109) and STS (+0.010) at 1/18th the size — and this checkpoint is also the WiSE-FT interpolation anchor that made that release possible. That final model is now on the official FinMTEB leaderboard: STS #2 of 18 (top embedding model on the board) and Summarization #2, see §2.

This card documents the full project journey — goal, every stage, every measured number — with this checkpoint's role in it.

1. The project goal

Build the strongest defensible finance embedder on a single ModernBERT-large (395M), evaluated on the full English FinMTEB benchmark (35 tasks / 7 types) against the published SOTA, Fin-E5 (finance-adapted e5-mistral-7B; paper). Target: win STS and Summarization, make Retrieval competitive. "Defensible": every stage trains behind a hard decontamination gate against all FinMTEB-EN eval sets.

2. Where the pipeline ended up (context for this checkpoint)

Final model results, full FinMTEB-EN 35 tasks (see the embed card for per-task detail):

Official FinMTEB leaderboard placement (EN board, 18 models, as of 2026-07-19): the final model ranks STS #2 (the top embedding model — only a lexical bag-of-words baseline scores higher; ahead of Fin-E5, voyage-3-large and NV-Embed v2), Summarization #2 (behind only voyage-3-large, with the board's single best FINDsum score, 0.745), and #11 overall. Only four models on the board hold two or more top-2 task-type finishes — voyage-3-large, text-embedding-3-large, Fin-E5, and this pipeline's 395M open-weights model.

And the journey milestone table:

3. This training run (from the actual logs)

Decontamination (before any training)

Every FinMTEB-English eval text was indexed with a word-shingle overlap index; corpus records overlapping any eval set, eval-source datasets, and test/dev splits were dropped; a hard assert_clean gate refuses to emit a corpus otherwise. Known-contaminated lineages (FiQA/ConvFinQA etc.) excluded outright. The corpus is not released (source licensing does not permit redistribution).

4. What DAPT does — and doesn't do (measured honestly)

Scored on FinMTEB-EN (15-task subset: 2 STS + 3 Summ + 10 Retrieval), mean pooling:

DAPT alone barely moves zero-shot embedding scores — an MLM objective doesn't reshape sentence geometry. Its measured value in this project came from two places:

1. Foundation for contrastive training. Stage 3 (270,238 synthetic + open pairs, positive-aware hard negatives mined with e5-large-v2, CachedMNRL scale 20 + CoSENT, 2 epochs on 2×3090) was trained from this checkpoint and reached Retrieval 0.569 / Summarization 0.625 (pure).
2. WiSE-FT interpolation anchor. Pure contrastive training collapsed STS 0.450 → 0.371, failing the pre-registered release gate (≥0.434 = Fin-E5 parity). The fix: w = α·contrastive + (1−α)·this_checkpoint:

* with chunk+mean-pool doc-side embedding. Note α=0.5 beats both endpoints on STS — the classic WiSE-FT effect. A finance-DAPT anchor made the interpolation safe: general knowledge returns without leaving the finance domain.

5. The rest of the pipeline (what was built on top)

• Stage 2 — synthetic pairs: an open-weights 35B MoE teacher (AWQ/vLLM) over SEC chunks → 115,238 filtered finance pairs incl. 16,696 doc↔summary positives + 16,700 mismatched score-0 summary negatives (the lever Fin-E5 never trained — its own adaptation regressed Summarization 0.528→0.480) + 1,452 term→definitions; blended with open general data to 270,238 pairs (59% finance).
• Long-doc discovery: chunk+mean-pool doc embedding = Summarization 0.120→0.305 with zero training (FNS docs average ~70k+ tokens; head-truncation was destroying signal).
• Stage 3 — contrastive: 2 epochs / 1,976 steps, effective 256 anchors/step, 255 in-batch negatives/device, lr 2e-5 cosine, bf16, ~8h on the same 2×3090 pair.

6. Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tok = AutoTokenizer.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")
model = AutoModelForMaskedLM.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")

text = "The company's [MASK] margin expanded 150 basis points year-over-year."
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
mask_idx = (inputs.input_ids == tok.mask_token_id).nonzero()[0, 1]
print(tok.decode(logits[0, mask_idx].topk(5).indices))

As a fine-tuning base (recommended use):

from transformers import AutoModel
encoder = AutoModel.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")  # drops MLM head

Native context 8,192 tokens (ModernBERT); DAPT windows were 1,024 tokens. English, finance-domain, Apache-2.0.

7. Limitations

• Not an embedding model out of the box — zero-shot sentence geometry ≈ the untrained base (see §4). Use the embed model for embeddings, or fine-tune this.
• English-only, finance-domain MLM; general-domain MLM quality may have drifted.
• Corpus not released (licensing); decontamination applies to FinMTEB-EN specifically.

Related

• FinModernBERT-embed-large-v1 — the final contrastive embedding model built from this checkpoint.
• FinModernBERT-pairs-sec-synthetic-v1 — the released 115k SEC-synthetic contrastive pair set used in Stage 3.
• ModernBERT-large — base model.
• FinMTEB — benchmark (paper).