gem-xlm-roberta

GEM-XLM-RoBERTa Legal 26G: A Cross-Lingual Greek-English Legal Language Model

Model Description

GEM-XLM-RoBERTa Legal 26G is an XLM-RoBERTa-base model pre-trained from scratch on a comprehensive 26GB bilingual corpus of Greek and English legal, parliamentary, and governmental text. This model leverages the advanced cross-lingual capabilities of the XLM-RoBERTa architecture to create superior bilingual legal understanding and cross-lingual transfer learning.

The model builds upon the XLM-RoBERTa architecture, which is specifically designed for multilingual and cross-lingual tasks, making it ideal for bilingual legal applications. Unlike standard RoBERTa, XLM-RoBERTa incorporates advanced cross-lingual pre-training techniques that enable better alignment between Greek and English legal concepts.

This model represents a significant advancement in cross-lingual legal language modeling, combining the rich legal traditions of Greek civil law and English common law in a unified multilingual framework. The model builds upon legal datasets including portions of the Pile of Law collection from Hugging Face, enhanced with comprehensive Greek legal corpora.

How to Get Started

You can use this model directly with the fill-mask pipeline:

from transformers import pipeline

# Load the model
fill_mask = pipeline(
    "fill-mask",
    model="novelcore/gem-xlm-roberta",
    tokenizer="novelcore/gem-xlm-roberta"
)

# Example in Greek
text_gr = "Ο κ. Μητσοτάκης <mask> ότι η κυβέρνηση σέβεται πλήρως τις αποφάσεις του Συμβουλίου της Επικρατείας."
predictions_gr = fill_mask(text_gr)
print("Greek predictions:", predictions_gr)

For downstream tasks:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# For cross-lingual legal document classification
tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-xlm-roberta")
model = AutoModelForSequenceClassification.from_pretrained("novelcore/gem-xlm-roberta")

# Process texts in both languages with shared representations
greek_text = "Το Συνταγματικό Δικαστήριο αποφάσισε..."
english_text = "The Constitutional Court decided..."

# The model can handle both with unified understanding

Training Data

The model was pre-trained on a comprehensive 26GB bilingual corpus comprising 60.3% Greek legal content (13.85GB) and 39.7% English legal content (9.12GB), optimized for cross-lingual legal understanding through the XLM-RoBERTa architecture.

Greek Legal Corpus (13.85 GB - 60.3%)

English Legal Corpus (9.12 GB - 39.7%)

Note: English legal datasets partially sourced from the Pile of Law collection on Hugging Face.

Cross-Lingual Training Strategy

The XLM-RoBERTa architecture enables superior cross-lingual learning through:

• Shared multilingual representations across Greek and English legal concepts
• Cross-lingual alignment of parallel legal documents (EU legal texts in both languages)
• Legal concept transfer between civil law (Greek) and common law (English) systems
• Enhanced multilingual tokenization optimized for both Latin and Greek scripts

Training Procedure

Model Architecture

The model uses the XLM-RoBERTa-base architecture with the following configuration:

• Hidden Size: 768
• Attention Heads: 12
• Hidden Layers: 12
• Parameters: ~125M
• Max Position Embeddings: 514
• Vocabulary Size: 50,264 (Optimized for Greek-English bilingual legal text)

Preprocessing

The text was tokenized using a custom multilingual SentencePiece tokenizer trained from scratch on the bilingual Greek-English legal corpus. The tokenizer uses a vocabulary of 50,264 tokens specifically optimized for cross-lingual legal terminology, enabling effective shared representations between Greek and English legal concepts.

The data was processed into fixed-size chunks of 512 tokens, with special attention to maintaining cross-lingual document alignment and contextual coherence across both languages.

Pre-training

The model was pre-trained from scratch for 150,000 steps on 8x NVIDIA H100 GPUs, using BFloat16 (bf16) mixed-precision for stability and speed. The training took approximately 24 hours and 27 minutes to complete.

The key hyperparameters used were:

• Learning Rate: 2e-4 (0.0002) with a linear warmup of 9,000 steps
• Batch Size: Effective batch size of 256 (per_device_train_batch_size: 128, gradient_accumulation_steps: 2)
• Optimizer: AdamW with standard parameters
• Weight Decay: 0.01
• Max Sequence Length: 512
• Max Steps: 150,000
• Warmup Steps: 9,000
• MLM Probability: 0.15
• Max Gradient Norm: 1.0

Training Results

The model achieved excellent performance metrics, demonstrating superior cross-lingual convergence:

• Final Training Loss: 0.7165
• Final Evaluation Loss: 0.660432
• Training Infrastructure: 8x NVIDIA H100 GPUs
• Total Training Steps: 150,000
• Total Training Time: 24 hours 27 minutes
• Train/Validation Split: 95%/5%
• Total Training Data: 26GB bilingual corpus