HelaBERT_Large

HelaBERT-Large

HelaBERT-Large is a BERT-based masked language model pre-trained from scratch on a large Sinhala text corpus. With approximately 110 million parameters, it is designed to produce contextual representations of Sinhala language text and can be used for downstream NLP tasks such as text classification, named entity recognition, semantic similarity, and information retrieval.

This work is described in the paper: HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head
arXiv:2608.22922

---

Model Details

---

Training Data

HelaBERT-Large was pre-trained on approximately 1.1 billion tokens (~26.5 million lines) of Sinhala text sourced from three datasets:

• MADLAD-400 — Sinhala subset of the multilingual document-level dataset
• CulturaX — Sinhala subset of the cleaned multilingual web corpus
• Custom Sinhala Corpus — A dataset compiled from Sinhala Wikipedia, Sinhala news articles, and Sinhala web crawl data

Data Preprocessing

The raw text was preprocessed through a multi-stage cleaning pipeline before training:

• Unicode NFC normalization and removal of zero-width characters (excluding ZWJ U+200D, which is required for correct Sinhala ligature rendering)
• Filtering to retain only lines containing Sinhala characters (U+0D80–U+0DFF), with a minimum line length of 5 characters
• Removal of non-Sinhala characters, retaining Sinhala script, digits, common punctuation, and ZWJ/ZWNJ
• Normalization of repeated punctuation, extra whitespace, unmatched brackets, and date-like numeric patterns
• Final text was combined into a single corpus and tokenized using a SentencePiece unigram tokenizer

---

Tokenizer

HelaBERT-Large uses a SentencePiece Unigram tokenizer trained on Sinhala text with a vocabulary size of 32,000. The tokenizer is not included in the HuggingFace tokenizer format and must be used via the sentencepiece library directly.

import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.load("tokenizer/unigram_32000_0.9995.model")

ids = sp.encode("ශ්‍රී ලංකාවේ අගනුවර කොළඹ වේ", out_type=int)
print(ids)

---

Training Configuration

---

Training Results

Training loss decreased from ~10.3 to ~2.26 over 6 epochs (~90,700 steps), with validation loss converging from ~7.5 to ~2.17, indicating no significant overfitting.

---

Hardware & Environmental Impact

CO₂ Estimate: 0.700 kW × 22.5 h × 0.387 kg CO₂/kWh ≈ 6.09 kg CO₂eq

Grid carbon intensity for the US Southeast (SRSO subregion) sourced from EPA eGRID 2022 (~0.387 kg CO₂/kWh), reflecting DigitalOcean's Atlanta data center. This estimate covers GPU power draw only and does not account for CPU, RAM, or system-level power consumption, so the actual footprint is moderately higher.

---

Usage

Masked Language Modeling (Fill-Mask)

from transformers import BertForMaskedLM
import sentencepiece as spm
import torch

sp = spm.SentencePieceProcessor()
sp.load("tokenizer/unigram_32000_0.9995.model")

model = BertForMaskedLM.from_pretrained("HelaBERT-Large")
model.eval()

sentence = "ශ්‍රී ලංකාවේ [MASK] අගනුවර කොළඹ වේ"
mask_id = sp.piece_to_id("[MASK]")

parts = sentence.split("[MASK]")
input_ids = sp.encode(parts[0], out_type=int) + [mask_id] + sp.encode(parts[1], out_type=int)
input_ids = torch.tensor([input_ids])

with torch.no_grad():
    logits = model(input_ids).logits

mask_index = (input_ids == mask_id).nonzero(as_tuple=True)[1]
top5 = torch.topk(logits[0, mask_index], 5, dim=-1)

print("Top 5 predictions for [MASK]:")
for token_id in top5.indices[0]:
    print(sp.id_to_piece(token_id.item()))

Sentence Embeddings

from transformers import BertModel
import sentencepiece as spm
import torch

sp = spm.SentencePieceProcessor()
sp.load("tokenizer/unigram_32000_0.9995.model")

model = BertModel.from_pretrained("HelaBERT-Large", add_pooling_layer=False, ignore_mismatched_sizes=True)
model.eval()

def embed(text):
    ids = torch.tensor([sp.encode(text, out_type=int)])
    mask = (ids != sp.pad_id()).unsqueeze(-1)
    with torch.no_grad():
        out = model(ids)
    return ((out.last_hidden_state * mask).sum(dim=1) / mask.sum(dim=1)).squeeze(0)

embedding = embed("කෘත්‍රිම බුද්ධිය ශ්‍රී ලංකාවේ අනාගතය වෙනස් කරයි")
print(embedding.shape)  # torch.Size([768])

Semantic Similarity

import torch.nn.functional as F

e1 = embed("කෘත්‍රිම බුද්ධිය අනාගතය වෙනස් කරයි")
e2 = embed("කෘත්‍රිම බුද්ධිය අනාගතයට බලපායි")

score = F.cosine_similarity(e1, e2, dim=0)
print(f"Similarity: {score.item():.4f}")

Similarity score interpretation:

---

Limitations

• Language scope: HelaBERT-Large is trained exclusively on Sinhala text. It does not support multilingual inference or cross-lingual transfer.
• Tokenizer compatibility: The SentencePiece tokenizer is not natively integrated with the HuggingFace AutoTokenizer API. Manual tokenization is required.
• Training data bias: The corpus includes web-crawled content, which may contain informal language, spelling inconsistencies, or undesirable content that was not fully filtered.
• Downstream evaluation: HelaBERT-Large has not yet been formally benchmarked on labelled Sinhala NLP tasks (e.g., NER, classification). Reported similarity scores are from qualitative inference tests only.
• Dialectal variation: The model may underrepresent dialectal and colloquial Sinhala, as the corpus skews toward written, formal text.

---

Citation

If you use HelaBERT-Large in your research or work, please cite:

@misc{ekanayake2026helabertenhancingsinhalalanguage,
      title={HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head}, 
      author={Thisen Ekanayake and Nisansa de Silva},
      year={2026},
      eprint={2608.22922},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.22922}, 
}

---

Acknowledgements

Pre-training data sourced from MADLAD-400, CulturaX, Sinhala Wikipedia, Sinhala news sources, and Sinhala web crawl data.