HalleluBERT_base

HalleluBERT: Let every token that has meaning bear its weight

HalleluBERT is a family of RoBERTa-based Modern Hebrew language models pre-trained from scratch on ~49.1 GB of deduplicated Hebrew web text (HeDC4 / HeRo corpus) and Hebrew Wikipedia. The models aim to provide the first fully converged Hebrew RoBERTa encoder family, including a large variant, and to push state-of-the-art performance on core Hebrew benchmarks.

We release two variants:

• HalleluBERT-base: 126M parameters (fp32)
• HalleluBERT-large: 357M parameters (fp32)

Model Details

Downstream Evaluation

We evaluate HalleluBERT on three Hebrew benchmarks (following the HeRo suite, restricted to NER + sentiment):

• NER (BMC split 1): micro-F1
• NER (NEMO², token-level): micro-F1
• Sentiment (SMCD, deduplicated): macro-F1

We select the best configuration by validation performance and report the best score out of 10 runs on the official test sets.

🧪 Evaluation Results

Legend: Bold = best, <u>underline = second-best</u> within each model size group.

Fairseq Checkpoint

Get the fairseq checkpoint here.

Citation

If you use HalleluBERT in your research, please cite the corresponding paper (replace with your final bib entry if you already have one):

@inproceedings{schmitt-2026-hallelubert,
  title = {HalleluBERT: Let Every Token That Has Meaning Bear Its Weight},
  author = {Schmitt, Raphael},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {3022--3030},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/3qdqexx4e9i2}
}

📜 License

MIT License