ogbert-v1-mlm

OGBert v1 MLM

OGBert (OpenGloss BERT) is a ModernBERT-based masked language model pretrained on the OpenGloss synthetic encyclopedic dictionary. Despite being trained on a relatively small corpus of ~160M words (435K dictionary entries), the model achieves strong performance on definition understanding and domain-specific terminology.

The training corpus contains definitions across 16 domains (geography, mathematics, science, law, technology, philosophy, etc.) and 11 reading levels (kindergarten through PhD).

Model Description

• Model type: ModernBERT for Masked Language Modeling
• Language: English
• License: Apache 2.0
• Parameters: ~38M
• Context length: 1024 tokens
• Training data: mjbommar/ogbert-v1-mlm
• Built with: Transformers v5.0

Architecture

The model uses ModernBERT's hybrid attention pattern with full attention every 3 layers and sliding window attention in between, enabling efficient processing of long sequences.

Intended Uses

Primary Use Cases

• Fill-mask tasks: Predicting masked tokens in dictionary/definition text
• Feature extraction: Generating embeddings for downstream tasks
• Fine-tuning base: Starting point for domain-specific models

Domain Strengths

The model shows strong performance on:

• Geography (0.44 loss) - Place names and geographic terminology
• Mathematics (0.56 loss) - Mathematical and symbolic language
• Society (0.60 loss) - Social science terminology
• Science (0.63 loss) - Natural science terminology

How to Use

Fill-Mask Pipeline

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="mjbommar/ogbert-v1-mlm")
result = fill_mask("A molecule is the smallest <|mask|> of a chemical compound.")
print(result)

Example outputs:

Feature Extraction

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("mjbommar/ogbert-v1-mlm")
model = AutoModel.from_pretrained("mjbommar/ogbert-v1-mlm")

text = "Photosynthesis is the process by which plants convert light into energy."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)  # Mean pooling

print(embeddings.shape)  # torch.Size([1, 384])

Masked Language Modeling

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("mjbommar/ogbert-v1-mlm")
model = AutoModelForMaskedLM.from_pretrained("mjbommar/ogbert-v1-mlm")

text = "A molecule is the smallest <|mask|> of a chemical compound."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    mask_idx = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
    predictions = outputs.logits[0, mask_idx].softmax(dim=-1)
    top_tokens = predictions.topk(5)

for score, idx in zip(top_tokens.values[0], top_tokens.indices[0]):
    print(f"{tokenizer.decode(idx)}: {score:.4f}")
# Output:
#   unit: 0.6509
#   part: 0.1069
#   component: 0.0541
#   form: 0.0294
#   portion: 0.0243

Training Details

Training Data

• Dataset: mjbommar/ogbert-v1-mlm
• Source: OpenGloss v1.1 Dictionary
• Domains: 16 (Geography, Mathematics, Science, Law, Technology, Philosophy, etc.)
• Reading levels: 11 (Kindergarten through PhD)

Training Procedure

Hyperparameters

Training Infrastructure

• Framework: Transformers + Accelerate
• Hardware: Single GPU

Final Training Metrics

From step 5000 (final checkpoint):

Loss Stability (Final 1000 Steps)

Evaluation Results

Clustering Performance

Retrieval Performance

Word Similarity (SimLex-999)

Training Convergence

Limitations

1. Word similarity: The model achieves relatively low word similarity scores (SimLex 0.29). MLM pretraining optimizes for categorical boundaries rather than pairwise similarity. For tasks requiring fine-grained similarity, consider contrastive fine-tuning.

2. Domain coverage: Performance varies by domain. Arts and history show higher loss (0.77-0.84) compared to geography and mathematics (0.44-0.56).

3. English only: The model is trained exclusively on English text.

Related Models

• Base architecture: answerdotai/ModernBERT-base
• Training data: mjbommar/opengloss-v1.1-dictionary

Citation

If you use this model, please cite the OpenGloss paper:

@misc{bommarito2025opengloss,
  title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph},
  author={Bommarito, Michael J., II},
  year={2025},
  eprint={2511.18622},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2511.18622}
}

License

This model is released under the Apache 2.0 license.