judaic-semantic-tokenizer-32k-v1

Judaic Semantic Tokenizer 32K v1

Production-oriented Unigram/Metaspace tokenizer for Biblical and Rabbinic Hebrew, Babylonian and Jerusalem Talmud Aramaic, medieval and modern Hebrew, limited Judeo-Arabic, and modern Hebrew/Latin code-switching.

This is a tokenizer artifact, not an embedding encoder. It is incompatible with checkpoints trained using the earlier 24K vocabulary; new encoder checkpoints must be initialized with this 32K vocabulary.

Training coverage

• source database scanned: 8,010 books and 3.242 billion raw characters;
• training exposure: 450,005,532 normalized characters;
• 2,022,860 weighted training examples;
• 1,069,509 unique normalized texts;
• 13 linguistic strata balanced by normalized characters rather than row count;
• 18,931 held-out evaluation examples;
• 796,277,866-byte training corpus, SHA-256 f2a8fe1755e82f139237c10327868b4ed9f6a68357d1610f4832c3d9f5a1c5f2.

The raw training corpus is not redistributed because the underlying books have source-dependent rights.

Evaluation

All seven acceptance gates passed. See evaluation.json and corpus_report.json for per-stratum metrics and sampling details.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ArieLLL123/judaic-semantic-tokenizer-32k-v1")
query = tokenizer("[QUERY] חיפוש בניסוח חופשי", truncation=True, max_length=512)
passage = tokenizer("[PASSAGE] טקסט תורני לאינדוקס", truncation=True, max_length=512)

Nine special tokens are included: [PAD], [UNK], [CLS], [SEP], [MASK], [QUERY], [PASSAGE], [שאילתה], and [קטע].

Normalization

The training pipeline uses NFKC, strips HTML markup while preserving visible text, removes Hebrew diacritics and invisible/bidi controls, canonicalizes punctuation, and preserves Latin letters for modern code-switching. Reproducible preprocessing and training code are included under training_code/; preprocessing.py is provided for runtime use.

<!-- judaic-semantic-rights:start -->

Project contribution and license

The tokenizer vocabulary, normalization interface, evaluation, packaging, and project-authored metadata were produced by the Judaic Semantic Embedding project within Otzaria. This statement concerns those artifacts, not authorship of source texts.

Project-authored model/tokenizer artifacts are licensed under CC BY-NC-SA 4.0. Any included project-authored code uses the Personal Use License 1.0 in LICENSE.md; third-party files retain their own terms.

<!-- judaic-semantic-rights:end -->