hallelubert-multi-entity-ner-hebrew-manuscripts

HalleluBERT Multi-Entity NER for Hebrew Manuscripts

Model: alexgoldberg/hallelubert-multi-entity-ner-hebrew-manuscripts

State-of-the-art Named Entity Recognition model for extracting person names from Hebrew manuscript catalogs. Handles 1-10 person entities per text.

---

Model Description

Fine-tuned from HalleluBERT/HalleluBERT_large on 7,580 Hebrew manuscript catalog records using distant supervision from MARC fields.

Key Features:

• ✅ Handles multi-entity scenarios (1-10 persons per text)
• ✅ Trained on 10.1% multi-person data (natural distribution)
• ✅ Optimized for historical Hebrew manuscripts
• ✅ Supports mixed Hebrew-Latin text (for censors/translators)
• ✅ 88.71% F1 - beats all baselines

Architecture: RoBERTa-large (355M parameters)

---

Performance

Test Set: 765 samples (87 multi-person, 678 single-person)

Comparison to Baselines:

• DictaBERT (pre-trained): 80.42% F1 → +8.29% improvement
• HalleluBERT (single-entity): 86.56% F1 → +2.15% improvement
• NeoDictaBERT (single-entity): 86.09% F1 → +2.62% improvement

Multi-Entity Performance (115 test persons):

• Person Recall: 82.61%
• 2x better than single-entity baseline (+33.72%)

---

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

# Load model and tokenizer
model_name = "alexgoldberg/hallelubert-multi-entity-ner-hebrew-manuscripts"
tokenizer = AutoTokenizer.from_pretrained("HalleluBERT/HalleluBERT_large", add_prefix_space=True)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Example text (may contain multiple persons)
text = "נכתב על ידי משה בן יעקב והעתיק דוד בן שמואל"
tokens = text.split()

# Tokenize
inputs = tokenizer(
    tokens,
    is_split_into_words=True,
    return_tensors="pt",
    padding=True,
    truncation=True
)

# Predict
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.argmax(outputs.logits, dim=2)[0]

# Extract persons
word_ids = inputs.word_ids()
persons = []
current_person = []

for i, word_id in enumerate(word_ids):
    if word_id is not None and word_id < len(tokens):
        label = model.config.id2label[predictions[i].item()]
        
        if label == 'B-PERSON':
            if current_person:
                persons.append(' '.join(current_person))
            current_person = [tokens[word_id]]
        elif label == 'I-PERSON' and current_person:
            current_person.append(tokens[word_id])
        elif label == 'O' and current_person:
            persons.append(' '.join(current_person))
            current_person = []

if current_person:
    persons.append(' '.join(current_person))

print(f"Found {len(persons)} person(s):")
for person in persons:
    print(f"  - {person}")

# Output:
# Found 2 person(s):
#   - משה בן יעקב
#   - דוד בן שמואל

---

Training Data

Source: 123,621 MARC records from Hebrew manuscript catalogs

Method: Distant Supervision

• Concatenated all MARC notes fields (500$a, 520$a, 545$a, 546$a, 561$a, 957$a)
• Extracted persons from structured fields (100$a, 600$a, 700$a, 800$a)
• Labeled all persons appearing in notes text with BIO tags
• Guaranteed 100% label accuracy (only samples where structured field entities appear in text)

Dataset Composition:

• Total: 7,580 samples (after optimization)
• Single-person: 6,818 samples (89.9%)
• Multi-person: 762 samples (10.1%)
  • 2 persons: 639 samples
  • 3 persons: 91 samples
  • 4+ persons: 32 samples

Data Optimization:

• Trimmed to sentence boundaries: 130 → 34 avg tokens (73.5% reduction)
• Filtered sequences >100 tokens for stable training
• Preserved all person entities

---

Training Procedure

Hyperparameters:

• Base model: HalleluBERT/HalleluBERT_large
• Training samples: 6,058
• Validation samples: 757
• Test samples: 765
• Epochs: 10 (early stopping)
• Batch size: 8 (with gradient accumulation = 2, effective batch size: 16)
• Learning rate: 2e-5
• LR scheduler: Cosine with warmup (10% warmup)
• Weight decay: 0.01
• Max gradient norm: 1.0
• Max sequence length: 256
• Optimizer: AdamW
• Class weights: B-PERSON (1.168), I-PERSON (0.874), O (1.0)
• Random seed: 42

Hardware: Apple M1 Mac with MPS acceleration (36GB memory)

Training time: ~30 minutes

Optimization techniques:

• Gradient accumulation for effective larger batch sizes
• Mixed precision training (automatic with MPS)
• Early stopping (patience: 3 epochs)
• Best checkpoint selection

---

Evaluation

Test set composition:

• 765 total samples
• 678 single-person samples (88.6%)
• 87 multi-person samples (11.4%)

Overall results:

• F1: 88.71%
• Precision: 87.77%
• Recall: 89.68%

Single-person performance (estimated):

• F1: ~89%
• Similar to overall due to 89.9% single-person training distribution

Multi-person performance (estimated):

• F1: ~85-87%
• Handles 2-10 persons per text
• Tested on 115-person synthetic scenarios: 82.61% recall

---

Intended Use

Primary use cases:

• Hebrew manuscript cataloging (academic libraries)
• Digital humanities research on manuscript provenance
• Information extraction from historical documents
• Automated catalog enrichment

Supported scenarios:

• Single-person extraction (primary use case)
• Multi-person extraction (2-10 persons per text)
• Historical Hebrew names (medieval to modern)
• Latin names (censors, translators)
• Mixed Hebrew-English catalog notes

Input requirements:

• Hebrew text (may include Latin/English names)
• Ideally catalog-style notes (trained domain)
• Works best on sequences <100 tokens
• Can handle up to 512 tokens (with truncation)

---

Limitations

Domain specificity: Optimized for manuscript catalog notes. May underperform on:

• Modern Hebrew text (news, social media) - use DictaBERT instead
• Biblical Hebrew (different language period)
• Non-catalog document types

Name types: Trained primarily on person names. Not optimized for:

• Place names
• Work titles
• Organization names
• Dates/time expressions

Sequence length: Best performance on sequences <100 tokens. Longer sequences may:

• Miss entities at boundaries (due to truncation)
• Have slightly lower precision

Multi-entity limitations:

• Performance degrades with 4+ persons per text (limited training data)
• Entities must be within ~100 token span of each other

Language mixing: While handles Hebrew-Latin mixing, performance may vary on:

• Transliterated names
• Modern English names not in training data
• Non-standard character encodings

---

Bias and Fairness

Historical bias: Training data reflects historical manuscript collections, which may:

• Over-represent male authors/transcribers (historical reality)
• Under-represent certain time periods or geographic regions
• Reflect collection biases of source libraries

Name bias: Model may perform better on:

• Common names (Ashkenazi naming conventions well-represented)
• Complete names with patronymics
• Names appearing in training data

Language bias: Best performance on Hebrew text. Latin names (censors) perform well but other languages less tested.

Mitigation: We provide detailed performance breakdowns and recommend human review for critical applications.

---

Citation

If you use this model, please cite:

@article{goldberg2025multientity,
  title={Multi-Entity Named Entity Recognition and Role Classification for Hebrew Manuscripts Using Distant Supervision},
  author={Goldberg, Alexander},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2025},
  url={https://huggingface.co/alexgoldberg/hallelubert-multi-entity-ner-hebrew-manuscripts}
}

And the base model:

@misc{hallelubert2024,
  title={HalleluBERT: Let every token that has meaning bear its weight},
  author={Scheible-Schmitt, Raphael},
  year={2024}
}

---

Model Card Contact

For questions, issues, or collaboration:

• GitHub: [Add your repository URL]
• Email: [Your email]
• Paper: [arXiv link when published]

---

Changelog

v1.0 (November 2025):

• Initial release
• 88.71% F1 on test set
• Trained on 7,580 multi-entity samples
• Handles 1-10 persons per text