ZaraEmbed

ZaraEmbed

A verse embedder for the Standard Works of The Church of Jesus Christ of Latter-day Saints that finds the right verse first far more often than OpenAI on Book of Mormon, D&C and Pearl of Great Price chapters it has never seen. Ask a question in plain modern English, with or without the names in the verse, and ZaraEmbed returns the verse that answers it. It is ZaraBERTa fine-tuned for retrieval, so it reads Nephi, Zarahemla and LORD as the scripture-native words ZaraBERTa learned, not as fragments.

Trained by Brigham Young University-Idaho on public-domain text. Chapter headings, footnotes and Church study aids were not used.

ZaraAI family

Part of ZaraAI, BYU-Idaho's scripture-native model collection:

Purpose

These models are released for research on scripture and on domain-specific language models. They are also released as aids for personal scripture study. They are offered in the hope that they help readers find and understand the Book of Mormon, and come unto Christ.

What is new

1. It finds verses from chapters it never trained on

The held-out tests use 289 verses from 37 Book of Mormon, D&C and Pearl of Great Price chapters that no training pair touched. A separate language model, not one used for the training questions, wrote one question per verse in two styles, and the same 289 verses appear in both tests:

• no-names: a teenager's search that avoids the verse's names and distinctive words.
• story: a reader naming the speaker, audience or event.

Right verse first (top-1), both models embedding bare verse text:

On the Book of Mormon questions alone (149 of the 289), ZaraEmbed leads 103 vs 58 on no-names and 98 vs 81 on story.

For an open-weights reference, bge-large-en-v1.5 places 63 no-names and 78 story questions first.

2. On the human-written benchmark

The 152-question benchmark holds human-written study questions across all five volumes. Here OpenAI leads overall, and the Restoration-scripture questions are a tie:

ZaraEmbed's strength is the Restoration scripture it was built for. Paired with ZaraRerank, the pipeline moves ahead of OpenAI on the benchmark's Restoration questions too (47 vs 38 for OpenAI on bare verses, 41 with chapter context); see that card.

3. Trained on both bare verses and verses with chapter context

Half of the documents seen in training were shown as "chapter summary + verse", so the model accepts either form. The summaries are retrieval-oriented descriptions generated for this project, not Church chapter headings. All numbers on this card use bare verses.

Usage

Mean-pool last_hidden_state over real tokens, then L2-normalise. Prefix every text with a single space so verse-initial words tokenize like mid-sentence words. Questions use up to 160 tokens; verses up to 256.

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("BYU-Idaho/ZaraEmbed")
model = AutoModel.from_pretrained("BYU-Idaho/ZaraEmbed", add_pooling_layer=False).eval()

def embed(texts, max_length):
    batch = tok([" " + t for t in texts], padding=True, truncation=True,
                max_length=max_length, return_tensors="pt")
    with torch.no_grad():
        hidden = model(**batch).last_hidden_state
    mask = batch["attention_mask"].unsqueeze(-1).to(hidden.dtype)
    pooled = (hidden * mask).sum(1) / mask.sum(1)
    return F.normalize(pooled, dim=-1)

question = embed(["Why does helping other people count as serving God?"], max_length=160)
verses = embed([
    "And behold, I tell you these things that ye may learn wisdom; that ye may learn that when ye are in the service of your fellow beings ye are only in the service of your God.",  # Mosiah 2:17
    "And Cainan lived seventy years, and begat Mahalaleel:",  # Genesis 5:12
], max_length=256)
print((question @ verses.T).tolist())
# [[0.7151, -0.0116]]  -> Mosiah 2:17 scores far above the unrelated verse (observed on CPU)

For search, embed all verses once, then take the top 100 by cosine similarity and pass them to ZaraRerank.

Model details

• Architecture: RobertaModel, 24 layers, hidden 1024, cased, vocabulary 52,176 (ZaraBERTa's), 1024-d embeddings.
• Base: BYU-Idaho/ZaraBERTa.
• Objective: symmetric InfoNCE with in-batch negatives, temperature 0.05.
• Training: 3 epochs, batch 1,024 through gradient caching, lr 3e-5, bf16, about 5.6 hours on one NVIDIA GB10. This checkpoint is epoch 3, from a single training seed.
• 270,875 training pairs:
  • 233,856 question-verse pairs, synthetic questions generated by large language models: 8 plain-language questions per Restoration verse; 8 story-frame questions per Restoration verse, written from the chapter summary and neighbouring verses; earlier synthetic questions; and 1,594 name-disambiguation questions.
  • Verse-summary, neighbouring-verse and parallel-passage pairs.
• Hard negatives mined by an earlier ZaraEmbed version, with NV-Retriever positive-aware filtering (negatives scoring above 95% of the gold verse are dropped), plus a same-chapter hard negative for each story-frame question.
• pooling.json records the pooling settings and the per-epoch training history.

Intended uses

• Semantic search over the Standard Works from modern-English questions.
• First-stage retrieval for ZaraRerank or for a retrieval-augmented assistant.
• Verse-to-verse similarity and finding related passages.

Limitations and responsible use

• These models are study aids, not doctrinal authorities, and they are not an official publication of The Church of Jesus Christ of Latter-day Saints.
• Retrieval scores measure textual relevance, not interpretation; read every verse in its context.
• Results come from a single training seed, and the held-out questions were written by a language model.

Provenance and license

Base model roberta-large by Meta AI, MIT license, through ZaraBERTa; that notice applies to any redistribution of these weights. Training text: public-domain editions of the Old Testament, New Testament, Book of Mormon, Doctrine and Covenants and Pearl of Great Price. Training questions and chapter summaries were generated by language models over that text. Chapter headings, footnotes and Church study aids were not used. Trained and evaluated by Brigham Young University-Idaho, 2026. Released as open weights under the MIT license.

Citation

@misc{zaraembed2026,
  title  = {ZaraEmbed: a scripture-native verse embedder for the Standard Works},
  author = {Vallejo, Ron and Brigham Young University-Idaho},
  year   = {2026},
  url    = {https://huggingface.co/BYU-Idaho/ZaraEmbed}
}