gem-modernbert

GEM-ModernBERT HQ Legal: A Greek Legal Language Model with Advanced Optimization

Model Description

GEM-ModernBERT HQ Legal is a ModernBERT-base model pre-trained from scratch on a strategically curated 21GB corpus of Greek legal, parliamentary, and governmental text. This model leverages ModernBERT's cutting-edge architectural innovations including Flash Attention 2, StableAdamW optimizer, 1024-token context length, and advanced memory optimization techniques to deliver superior performance on Greek legal document understanding tasks.

Building upon our proven quality-based data repetition strategy, this model incorporates ModernBERT's state-of-the-art training methodology with 30% masking probability, trapezoidal learning rate scheduling, and optimized batch sizing for enhanced convergence and performance. The model is specifically designed to handle longer legal documents with its extended 1024-token context window while maintaining computational efficiency through advanced optimization techniques.

This model represents the culmination of our Greek legal language modeling research, combining domain expertise with the latest architectural advances in transformer-based language models. It has been optimized for downstream tasks such as Named Entity Recognition (NER), Text Classification, and Question Answering within the complex legal domain.

How to Get Started

You can use this model directly with the fill-mask pipeline:

from transformers import pipeline

# Load the model
fill_mask = pipeline(
    "fill-mask",
    model="novelcore/gem-modernbert-hq-legal",
    tokenizer="novelcore/gem-modernbert-hq-legal"
)

# Example from a legal context with longer sequence support
text = "Σύμφωνα με το άρθρο 15 του Συντάγματος, η <mask> των δικαιωμάτων του ανθρώπου αποτελεί βασική υποχρέωση του κράτους στο πλαίσιο της δημοκρατικής πολιτείας."

# Get predictions
predictions = fill_mask(text)
print(predictions)

For downstream tasks:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# For legal document classification with extended context
tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-modernbert-hq-legal")
model = AutoModelForSequenceClassification.from_pretrained("novelcore/gem-modernbert-hq-legal")

# The model supports up to 1024 tokens for longer legal documents

Training Data

The model was pre-trained on the same comprehensive corpus of Greek text used in our previous models, employing our proven quality-based data repetition strategy that increases exposure to higher-quality legal content. The original 16.75GB corpus was expanded to 21.12GB through strategic repetition, now processed with 1024-token sequences for enhanced context understanding.

Quality-Based Data Repetition Strategy

Enhanced Context Processing

With 1024-token sequences, this model can process:

• Complete legal articles without truncation
• Full court decisions with extended reasoning
• Complex legislative texts with multiple references
• Parliamentary debates with comprehensive context

Training Procedure

Model Architecture

The model uses the ModernBERT-base architecture with the following configuration:

• Hidden Size: 768
• Attention Heads: 12
• Hidden Layers: 12
• Parameters: ~139M
• Max Position Embeddings: 1024
• Vocabulary Size: 50,373
• Flash Attention 2: Enabled
• Context Length: 1024 tokens (2x longer than previous models)

Key Architectural Advantages

ModernBERT's innovations provide significant benefits for legal text processing:

1. Flash Attention 2: Memory-efficient attention computation for longer sequences
2. Extended Context: 1024-token sequences capture complete legal documents
3. StableAdamW Optimizer: Enhanced training stability and convergence
4. Optimized MLM: 30% masking probability for improved representation learning
5. Advanced Memory Management: Optimized CUDA memory allocation for large batches

Preprocessing

The text was processed into 1024-token chunks using ModernBERT's tokenizer (vocabulary: 50,373 tokens), providing excellent coverage of Greek legal terminology while maintaining compatibility with the base architecture.

Higher-quality sources were strategically repeated during the data preparation phase, with sequences now capturing much more context per training example.

Pre-training

The model was pre-trained from scratch for 150,000 steps on 8x NVIDIA H100 80GB GPUs, using BFloat16 (bf16) mixed-precision with advanced optimization techniques. The training took approximately 97 hours and 9 minutes to complete.

Key Training Optimizations

Batch Size Optimization:

• Per-device batch size: 16 (optimized for H100 memory)
• Gradient accumulation steps: 8
• Effective batch size: 1,024 (16 × 8 × 8 GPUs)
• Context length: 1024 tokens per sequence

StableAdamW Configuration:

• Learning Rate: 0.0002 (conservative for stable convergence)
• Weight Decay: 0.1
• Adam Beta1: 0.9
• Adam Beta2: 0.95
• Adam Epsilon: 1e-08
• Gradient Clipping: 1.0
• Epsilon Mode: element_wise

Advanced Learning Rate Schedule:

• Schedule Type: Polynomial decay with trapezoidal warmup
• Warmup Steps: 9,000
• Decay Power: 0.5 (square-root decay)
• Max Steps: 150,000

ModernBERT Specifications:

• MLM Probability: 0.30 (higher than traditional 15%)
• Max Sequence Length: 1024
• Flash Attention 2: Enabled with optimizations
• Memory Optimization: Advanced CUDA allocation strategies

Training Results

The model achieved excellent performance metrics:

• Final Training Loss: 0.7648
• Final Evaluation Loss: 0.7751
• Training Infrastructure: 8x NVIDIA H100 80GB GPUs
• Total Training Steps: 150,000
• Total Training Time: 97 hours 9 minutes
• Train/Validation Split: 90%/10%
• Effective Training Data: 21.12GB (with quality-based repetition)
• Context Length: 1024 tokens per sequence

Advanced Training Infrastructure

The model was trained with cutting-edge optimizations:

Flash Attention 2 Optimizations:

FLASH_ATTENTION_FORCE_FP16: "0"        # Use bfloat16
FLASH_ATTENTION_SKIP_RESHAPE: "1"      # Skip unnecessary reshapes
FLASH_ATTENTION_CAUSAL: "0"            # Non-causal for BERT
FORCE_FLASH_ATTENTION: "1"             # Force Flash Attention usage

Memory Optimization:

PYTORCH_CUDA_ALLOC_CONF: "max_split_size_mb:256,roundup_power2_divisions:16,expandable_segments:True,garbage_collection_threshold:0.8"

Distributed Training:

• Backend: NCCL with extended timeout configurations
• Mixed Precision: BFloat16 for optimal H100 performance
• Evaluation Frequency: Every 5,000 steps
• Checkpointing: Every 5,000 steps
• Logging: Every 250 steps

Key Innovations

ModernBERT Architecture Benefits

1. Extended Context Window: 1024 tokens vs 512 in previous models
2. Flash Attention 2: Memory-efficient attention for longer sequences
3. StableAdamW Optimizer: Enhanced training stability and convergence
4. Higher MLM Probability: 30% masking for improved representation learning
5. Trapezoidal LR Schedule: Optimized learning rate progression

Quality-Based Data Repetition

Consistent with our previous models:

1. Highest quality sources (legal dictionaries) repeated 4x
2. Medium-high quality sources (court reports) repeated 3x
3. Medium quality sources (EU legal texts) repeated 2x
4. Lower quality sources used once for diversity