hun-modernBERT-large-1k

Hun ModernBERT Large (Pretrained)

This model is a Hungarian-language, ModernBERT-large architecture-based, Masked Language Modeling (MLM) pretrained language model. The project's goal was to provide a high-performance Hungarian base model that serves as an efficient starting point for downstream NLP tasks (e.g., text classification, NER, question answering).

Model Details

• Architecture: ModernBERT-large (ModernBertForMaskedLM)
• Parameters: ~390M (standard ModernBERT-large size)
• Context Length: 1024 tokens
• Tokenizer: magyar-nlp-szine-java/ModernBERT-base_hun_tokenizer
• Base Model: answerdotai/ModernBERT-large

Training Data

The model was trained on a mixture of the following predominantly Hungarian datasets:

1. magyar-nlp-szine-java/FULL_LABELED_PRETRAIN_shuffled_2
2. magyar-nlp-szine-java/europarl_hun
3. FineWeb-HQ2 (Hungarian subset only)

Total Tokens: 4,417,679,836 tokens. The data was packed to a context length of 1024 tokens and shuffled prior to training (seed=142).

Training Procedure

The model was pretrained with the following hyperparameters and configurations:

• Task: Masked Language Modeling (MLM)
• Masking Probability: 30% (mlm_probability=0.30)
• Optimizer: AdamW (fused implementation, adam_epsilon=1e-6)
• Learning Rate: 1e-4, with cosine scheduling and 1% warmup
• Batch Size: 8 (per device) × 8 (gradient accumulation) = 64 effective batch size per device
• Epochs: 2
• Precision: bfloat16 (BF16) and tf32 enabled
• Hardware: 1x NVIDIA GeForce RTX 5070 Ti
• Additional Techniques: Gradient checkpointing (non-reentrant), weight decay (0.01), max gradient norm: 1.0

Evaluation Results

At the end of training (epoch 2), the following loss metrics were recorded:

Note: The lower eval loss is typical and can be attributed to regularization techniques (e.g., weight decay, dropout) and the specific characteristics of the evaluation split.

Environmental Impact

• Hardware: 1x NVIDIA GeForce RTX 5070 Ti
• Training Duration: ~209 hours
• Estimated Energy Consumption: ~65 kWh (based on average GPU power draw under load)
• Estimated CO₂ Emissions: ~25 kg CO₂eq (highly dependent on the local energy grid's carbon intensity).

Note: For precise carbon footprint tracking in future runs, we recommend integrating tools like CodeCarbon directly into the training loop.

Intended Use

This is a base model, not an instruction-tuned chat model.

• Recommended Use: Fine-tuning for downstream tasks (e.g., via AutoModelForSequenceClassification, AutoModelForTokenClassification), generating text embeddings, or continued pretraining on Hungarian corpora.
• Not Recommended For: Direct, zero-shot text generation or chatbot applications, as the model is not optimized for instruction following or conversational alignment.

Limitations

• The model has a maximum context window of 1024 tokens. Inputs exceeding this length will be truncated.
• Performance is optimal on domains similar to the training data (general web text, news, legal/parliamentary documents). Domain-specific tasks (e.g., medical, technical) will require additional fine-tuning.
• The tokenizer handles Hungarian diacritics and special characters according to the specific design of magyar-nlp-szine-java/ModernBERT-base_hun_tokenizer.

Citation

If you use this model in your research or development, please cite the original ModernBERT work and this model:

@misc{modernbert,
  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year={2024},
  eprint={2412.13663},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}