multi-religion-bert

MultiReligionBERT

MultiReligionBERT is a domain-adapted multilingual BERT model produced by continued masked language modelling (MLM) pre-training on a 12-language Bible corpus. Starting from bert-base-multilingual-cased (178M parameters), the model was trained for 30,000 steps on 372,652 verses spanning European and African languages. It is designed for cross-lingual religious NLP tasks, including zero-shot transfer to African languages that have limited representation in general-purpose multilingual pre-training corpora.

A companion English-only model, ReligionBERT, covers monolingual English religious NLP with stronger performance on English-specific tasks.

---

Model Details

---

Intended Use

MultiReligionBERT is intended for multilingual and cross-lingual NLP tasks on religious and biblical text, including:

• Zero-shot cross-lingual classification of religious text for African languages
• Multilingual semantic similarity between Bible verses across languages
• Book and section classification of biblical passages in multiple languages
• Feature extraction for multilingual religious NLP pipelines
• Multilingual masked language modelling on religious corpora

It is particularly suited to low-resource African language settings where general-purpose multilingual models such as mBERT have near-zero performance on domain-relevant tasks.

---

How to Get Started

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("LucasLicht/multi-religion-bert")
model = AutoModelForMaskedLM.from_pretrained("LucasLicht/multi-religion-bert")

# English
text_en = "For God so loved the [MASK] that he gave his only begotten Son."
# French
text_fr = "Car Dieu a tant aimé le [MASK] qu'il a donné son Fils unique."
# Amharic
text_am = "እግዚአብሔር ዓለሙን እጅግ [MASK] ስለ አፈቀረ።"

for text in [text_en, text_fr, text_am]:
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    masked_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
    logits = outputs.logits[0, masked_index]
    predicted_token = tokenizer.decode(torch.argmax(logits, dim=-1))
    print(f"{text[:40]}... => {predicted_token}")

For multilingual sentence embeddings or fine-tuning:

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("LucasLicht/multi-religion-bert")
model = AutoModel.from_pretrained("LucasLicht/multi-religion-bert")

sentences = [
    "The LORD is my shepherd; I shall not want.",       # English
    "L'Éternel est mon berger: je ne manquerai de rien.", # French
    "Yehova ndisafudzi wangu; handishayiwi chinhu.",     # Shona
]

for sent in sentences:
    inputs = tokenizer(sent, return_tensors="pt", truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    cls_embedding = outputs.last_hidden_state[:, 0, :]
    print(f"Embedding shape: {cls_embedding.shape}")  # [1, 768]

---

Training Details

Pre-Training Corpus

The corpus was sourced from the christos-c/bible-corpus repository. Twelve languages were selected to provide geographic and typological diversity across European and African language families.

Total pre-training corpus: 372,652 verses

A RAM-exhaustion issue during tokenisation of the full corpus was resolved by processing the corpus in 50,000-line chunks saved as HuggingFace dataset shards before concatenation.

Training Procedure

Training was conducted across multiple sessions with checkpoint recovery. Due to session interruptions, full logging begins at step 18,500. A PermanentDeleteCallback retained only the two most recent checkpoints to prevent storage exhaustion. All metrics were logged to Weights and Biases.

Training Loss Curve

---

Evaluation

Downstream Task Results

Three fine-tuning tasks were evaluated using automatically constructed datasets. All results are on held-out test sets. MultiReligionBERT is compared to its generic baseline (bert-base-multilingual-cased).

Semantic Similarity (21,994 verse pairs)

Book Classification (7,726 samples, 66 classes)

MultiReligionBERT outperforms mBERT by +3.88 accuracy points and +4.21 macro F1 points.

Extractive Question Answering (1,199 LLM-assisted examples, English)

Note: MultiReligionBERT underperforms mBERT on English extractive QA. Multilingual domain adaptation shifts the model's representations in ways that are counterproductive for English-only span extraction on a small dataset, consistent with the known risk of catastrophic forgetting of language-specific capabilities during continued pre-training.

---

Zero-Shot Cross-Lingual Transfer

MultiReligionBERT, mBERT, and XLM-RoBERTa were evaluated on zero-shot book classification applied to five African language Bible corpora. All models were fine-tuned exclusively on English classification data; no target-language training was performed.

Accuracy

Macro F1

MultiReligionBERT outperforms mBERT on 4 of 5 languages. The most notable result is on Ewe: mBERT predicts zero correct book labels across all 297 test samples, while MultiReligionBERT achieves 3.37% accuracy and 0.0187 macro F1. Ewe is a severely low-resource Gbe language with minimal representation in standard multilingual pre-training corpora, but it is covered by a New Testament translation included in the Bible pre-training data. This demonstrates that domain-specific corpus coverage of a low-resource language provides transfer signal that general-purpose multilingual pre-training at scale alone does not supply.

---

Datasets

---

Limitations

• Pre-training covers 12 languages. Performance on other religious traditions (Quran, Vedas, Buddhist sutras) and languages not in the pre-training corpus has not been evaluated.
• Ewe and Swahili are included only as evaluation languages (New Testament-only translations); their pre-training signal is limited to the NT portion.
• The model underperforms mBERT on English extractive QA due to domain shift during multilingual continued pre-training.
• The model inherits biases present in bert-base-multilingual-cased and may reflect translation-specific theological perspectives in the source Bible texts.
• Cross-lingual accuracy on all African languages remains low in absolute terms; results should be interpreted relative to mBERT rather than as production-ready performance.

---

Citation

If you use this model, please cite:

@misc{licht2025multireligionbert,
  title     = {ReligionBERT: Domain-Adaptive Pre-Training of BERT on Biblical Corpora for Religious NLP Tasks},
  author    = {Licht, Lucas},
  year      = {2025},
  note      = {Koforidua Technical University, Ghana. Model available at https://huggingface.co/LucasLicht/multi-religion-bert}
}

---

Related Model

---

Contact

For questions or collaboration, reach out via HuggingFace or GitHub: @Licht005