gem-electra

GEM-ELECTRA Legal: A Greek Legal Language Model

Model Description

GEM-ELECTRA Legal is an improved ELECTRA-base model pre-trained from scratch on a large, 17GB corpus of Greek legal, parliamentary, and governmental text. This second version incorporates refined training hyperparameters for enhanced performance and stability. It is designed for understanding the complex vocabulary and context of the legal domain in Greece and the EU.

This model was trained as part of a research project and has been optimized for downstream tasks such as Named Entity Recognition (NER), Text Classification, and Question Answering within the legal field. The ELECTRA architecture provides more efficient pre-training compared to masked language models like BERT by using a generator-discriminator approach.

How to Get Started

You can use this model directly with the fill-mask pipeline:

from transformers import pipeline

# Load the model
fill_mask = pipeline(
    "fill-mask",
    model="novelcore/gem-electra-legal",
    tokenizer="novelcore/gem-electra-legal"
)

# Example from a legal context
text = "Ο κ. Μητσοτάκης <mask> ότι η κυβέρνηση σέβεται πλήρως τις αποφάσεις του Συμβουλίου της Επικρατείας."

# Get predictions
predictions = fill_mask(text)
print(predictions)

For downstream tasks:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# For legal document classification
tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-electra-legal")
model = AutoModelForSequenceClassification.from_pretrained("novelcore/gem-electra-legal")

Training Data

The model was pre-trained on a comprehensive 17GB corpus of Greek text compiled from various legal and governmental sources. The corpus was carefully cleaned, UTF-8 encoded, and deduplicated to ensure high quality and diversity before training.

The composition of the training corpus is as follows:

Training Procedure

Model Architecture

The model uses the ELECTRA architecture with the following configuration:

• Discriminator Hidden Size: 768
• Discriminator Attention Heads: 12
• Discriminator Hidden Layers: 12
• Generator Size Fraction: 0.25 (192 hidden size generator)

Preprocessing

The text was tokenized using a custom ByteLevelBPE tokenizer trained from scratch on the Greek legal corpus. The tokenizer is uncased (does not distinguish between upper and lower case) and uses a vocabulary of 50,264 tokens.

The data was then processed into fixed-size chunks of 512 tokens, respecting document boundaries to ensure contextual coherence.

Pre-training

The model was pre-trained from scratch for 200,000 steps on 8x NVIDIA A100 40GB GPUs, using BFloat16 (bf16) mixed-precision for stability and speed. This second version incorporates improved hyperparameters for enhanced convergence and performance.

The key hyperparameters used were:

• Learning Rate: 1e-4 (0.0001) with a linear warmup of 12,000 steps
• Batch Size: Effective batch size of 3,840 (per_device_train_batch_size: 60, gradient_accumulation_steps: 8)
• Optimizer: AdamW with beta1=0.9, beta2=0.98, epsilon=1e-6
• Weight Decay: 0.01
• Max Sequence Length: 512
• Max Steps: 200,000
• Warmup Steps: 12,000
• Generator Loss Weight: 50.0
• Discriminator Loss Weight: 50.0

Training Results

The model achieved the following performance metrics:

• Final Training Loss: 0.0056
• Final Evaluation Loss: 0.0054
• Training Infrastructure: 8x NVIDIA A100 40GB GPUs
• Total Training Steps: 200,000