ogbert-110m-sentence

OGBert-110M-Sentence

A 110M parameter ModernBERT-based sentence embedding model for glossary and domain-specific text.

Related models:

• mjbommar/ogbert-110m-base - Base MLM model for fill-mask tasks

Model Details

Training

• Pretraining: Masked Language Modeling on domain-specific glossary corpus
• Dataset: mjbommar/ogbert-v1-mlm - derived from OpenGloss, a synthetic encyclopedic dictionary with 537K senses across 150K lexemes
• Checkpoint: Step 8K (selected for optimal downstream performance)
• Key finding: L2 normalization of embeddings is critical for clustering/retrieval performance

Performance

Document Clustering (ARI)

Evaluated on 80 domain-specific documents across 10 categories using KMeans clustering.

OGBert-110M-Sentence matches RoBERTa-base on clustering and beats MiniLM by 13%.

Document Retrieval (MRR)

Mean Reciprocal Rank for same-category document retrieval.

Word Similarity (SimLex-999)

Spearman correlation between model cosine similarities and human judgments.

OGBert-110M-Sentence achieves 5x better word similarity than BERT-base.

Summary vs Baselines

At the same size as BERT-base, OGBert-110M-Sentence achieves:

• 5x better word similarity (SimLex)
• Matches RoBERTa on clustering (ARI)
• 13% better clustering than MiniLM

Usage

Sentence-Transformers (Recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('mjbommar/ogbert-110m-sentence')
embeddings = model.encode(['your text here'])  # L2 normalized by default

Example - Domain Similarity:

sentences = [
    'The financial audit revealed discrepancies in the quarterly report.',
    'An accounting review found errors in the fiscal statement.',
    'The patient was diagnosed with acute respiratory infection.',
]
embeddings = model.encode(sentences)

The model correctly identifies higher similarity within the same domain.

Direct Transformers Usage

from transformers import AutoModel, AutoTokenizer
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained('mjbommar/ogbert-110m-sentence')
model = AutoModel.from_pretrained('mjbommar/ogbert-110m-sentence')

inputs = tokenizer('your text here', return_tensors='pt', padding=True, truncation=True)
outputs = model(**inputs)

# Mean pooling + L2 normalize (critical for performance)
mask = inputs['attention_mask'].unsqueeze(-1)
pooled = (outputs.last_hidden_state * mask).sum(1) / mask.sum(1)
embeddings = F.normalize(pooled, p=2, dim=1)

For Fill-Mask Tasks

Use mjbommar/ogbert-110m-base instead.

Citation

If you use this model, please cite the OpenGloss dataset:

@article{bommarito2025opengloss,
  title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph},
  author={Bommarito II, Michael J.},
  journal={arXiv preprint arXiv:2511.18622},
  year={2025}
}

License

Apache 2.0