Nomeda-MSA-64Kv1

Nomeda-MSA-64Kv1

High-performance Arabic MSA tokenizer artifacts for Nomeda.

This repository provides the release files required to run the tokenizer with the nomeda Python package.

Files

• Nomeda-MSA-64Kv1.tsv
• Nomeda-MSA-64Kv1.model
• Nomeda-MSA-64Kv1.vocab

Recommended Usage (Nomeda)

Install package:

pip install nomeda

Load by pretrained name:

from nomeda import NomedaMSATokenizer

tok = NomedaMSATokenizer.from_pretrained("nomeda-msa-64k")
r = tok.encode("العلم نور")
print(r.ids)
print(r.tokens)
print(tok.decode(r.ids))

Or load default:

from nomeda import NomedaMSATokenizer

tok = NomedaMSATokenizer.load_default()

Default behavior:

• from_pretrained("nomeda-msa-64k") resolves local paths first, then Hugging Face (if available).
• load_default() uses NOMEDA_DEFAULT_MODEL when set, otherwise defaults to nomeda-msa-64k.

Local Fallback

If you want to force local loading, set the model directory:

export NOMEDA_MODEL_DIR=/path/to/repo

from nomeda import NomedaMSATokenizer

tok = NomedaMSATokenizer.from_pretrained("nomeda-msa-64k")

Benchmark Snapshot

Dataset A: Arabic Quotes (short-form text)

• Source (HF): https://huggingface.co/datasets/HeshamHaroon/arabic-quotes
• Column: quote

Dataset B: Hindawi Arabic Sections (long-form text)

• Source (HF): https://huggingface.co/datasets/nomeda-lab/hindawi-arabic-sections/blob/main/test.parquet

Interpretation:

• On short Arabic quotes, MarBERT has a small fertility advantage, while nomeda-msa-64k delivers substantially higher throughput.
• On longer Hindawi passages, nomeda-msa-64k achieves the best fertility among all tested tokenizers.
• Across both datasets, nomeda-msa-64k combines strong segmentation efficiency with consistently high speed.

Reproducibility

• Benchmark script: test.py
• Text cleanup: Arabic diacritics removed before tokenization
• Main metrics: fertility, chars/token, throughput (words/sec), and efficiency percentage
• Evaluation mode: same preprocessing and measurement pipeline across all compared tokenizers

Fair-Use Comparison Note

Tokenizer comparisons are sensitive to normalization rules, preprocessing choices, corpus characteristics, library internals, and hardware. Use these results as a practical snapshot for the exact datasets and setup listed above.

Notes

• This release corresponds to the 64k MSA tokenizer (v1).
• Main runtime file is Nomeda-MSA-64Kv1.tsv.
• The tokenizer is loaded through Nomeda APIs, not through Transformers AutoTokenizer.

Citation

If you use this tokenizer in research or production reports, please cite this repository and include the benchmark setup details from this model card.