Downloads · 30 days
1.1K
30% of all-time downloads
modernbert-tr-base
ytu-ce-cosmos/modernbert-tr-base
modernbert-tr-base is a fill-mask model from ytu-ce-cosmos. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as apache-2.0.
<p align="center" <img src="assets/logo.webp" width="20%" alt="ModernBERT-TR" / </p <h1 align="center"ModernBERT-TR</h1
All-time downloads
3.7K
Public
Parameters
149M
5.5 GB on disk
Likes
16
Trending 1
Repo makeup
Click a slice to open those files.
.onnx1.9 GB · 76%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- apache-2.0
- Model type
- modernbert
- Access
- Public
- Created
- Jun 12, 2026
- Updated
- Jul 19, 2026
- SHA
- d16b9da8
Model card
From the Hugging Face model README
<p align="center">
<img src="assets/logo.webp" width="20%" alt="ModernBERT-TR" />
</p>
<h1 align="center">ModernBERT-TR</h1>
<p align="center">
<a href="https://cosmos-ytu.github.io/modernbert-tr/">Release blog</a> ·
<a href="https://huggingface.co/collections/ytu-ce-cosmos/modernbert-tr">Model collection</a>
</p>
A 150M-parameter Turkish encoder with an 8,192-token context window.
Model
ModernBERT-TR is a masked-language encoder based on the ModernBERT architecture with alternating global and local attention.
| Parameters | 149.4M |
| Context length | 8,192 tokens |
| Layers | 22 |
| Hidden width | 768 |
| Attention heads | 12 |
| GLU feed-forward width | 1,152 |
| Tokenizer | Cased WordPiece, 50,008 tokens |
Results
We evaluate ModernBERT-TR with task-specific fine-tuning on 28 TabiBench tasks, the fixed five-seed TrGLUE protocol, and frozen-encoder linear probing with encoder-fast-eval.
<p align="center"> <img src="assets/benchmark-totals.png" alt="ModernBERT-TR compared with TabiBERT, BERTurk, and mmBERT on TabiBench, TrGLUE, and fast-eval" /> </p>ModernBERT-TR leads the frozen-encoder evaluation and ranks second on TabiBench and TrGLUE.
<p align="center"> <img src="assets/capability-ranks.png" alt="Mean task rank by capability for ModernBERT-TR, TabiBERT, BERTurk, and mmBERT" /> </p>Mean rank across 47 tasks grouped by capability; lower is better. ModernBERT-TR leads classification, pairwise inference, and retrieval, and ties TabiBERT on acceptability.
Usage
from transformers import pipeline
unmask = pipeline(
"fill-mask",
model="ytu-ce-cosmos/modernbert-tr-base",
)
unmask("Türkiye'nin başkenti [MASK]'dır.")
Training
We pretrain from random initialization with packed sequences, dynamic masking, BF16, and StableAdamW. Our effective batch size is 1.05M tokens.
| Phase | Training |
|---|---|
| 1k pretraining | 200B tokens at length 1,024; 6B-token warmup to 5e-4, then constant learning rate; masking 0.30 |
| 1k short decay | 20B-token continuation from the 1k endpoint; decay from 5e-4 to 0; masking 0.15 |
| 1k low-LR continuation | Rollback to about 40B tokens; 100B tokens at 1e-4, masking 0.10 |
| 1k decay | 1k decay soup from 1k low-LR continuation and 1k short decay |
| 8k context extension | 50B-token continuation from the 1k endpoint at length 8,192; constant 2e-4; masking 0.30 |
| 8k terminal decay | 20B-token continuation on the 8k mix; decay from 2e-4 to 0; masking 0.10 |
The 1k pretraining mix is about 81.4% Turkish, 8.5% English, 10.0% code, and 0.1% parallel text. The 8k mix adds long-form Turkish PDF, legal, and long-document sources.
Revisions
| Revision | Contents |
|---|---|
main | ModernBERT-TR; 50% 1k decayed soup and 50% 8k terminal decay |
1k-no-decay | Direct 200B-token 1k pretraining endpoint |
1k-decay | 50% 1k low-LR continuation and 50% 1k short decay |
8k-no-decay | Direct 50B-token 8k context-extension endpoint |
License
Apache-2.0.
- Task
- Fill-Mask
- Library
- transformers
- Type
- modernbert
- License
- apache-2.0
- Languages
- tr
- Created
- Jun 12, 2026
- Updated
- Jul 19, 2026