Downloads · 30 days
0
nomeda-lab/Nomeda-Egyptian-16Kv1
Nomeda-Egyptian-16Kv1 is a machine learning model from nomeda-lab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for nomeda. The card lists the license as mit.
Egyptian Arabic (Masri) dialect plugin for the Nomeda tokenizer.
Downloads · 30 days
0
Access
Public
Updated Apr 28, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.tsv218 KB · 96%
From the Hugging Face model README
Egyptian Arabic (Masri) dialect plugin for the Nomeda tokenizer.
This plugin adds 8,257 dialect-specific tokens on top of the Nomeda-MSA-64Kv1 base vocabulary, bringing the combined vocab size to 72,257 tokens.
Training target was 16,000 plugin tokens. After filtering out tokens that already exist in the MSA base vocab, 8,257 unique Egyptian tokens survived — a higher-quality, non-redundant plugin.
Nomeda uses a base vocabulary + plugin vocabulary architecture:
This design guarantees:
| File | Description |
|---|---|
Nomeda-Egyptian-16Kv1.plugin.tsv | Plugin vocabulary (tab-separated: token, id, score) |
validation_report.json | ID conflict & coverage verification |
merge_stats.json | Training pipeline statistics |
validation_e2e.json | End-to-end round-trip & dialect word coverage |
pip install nomeda
from nomeda import NomedaMSATokenizer
# Load base MSA tokenizer
msa_tok = NomedaMSATokenizer.from_tsv("Nomeda-MSA-64Kv1/vocab/Nomeda-MSA-64Kv1.tsv")
# Load Egyptian plugin
msa_tok.load_plugin("Nomeda-Egyptian-16Kv1/Nomeda-Egyptian-16Kv1.plugin.tsv", "egy")
# Encode with explicit dialect
result = msa_tok.encode_with_dialect("مش عارف ايه اللي بيحصل", "egy")
print(result.tokens)
print(result.ids)
# Decode with dialect
print(msa_tok.decode_with_dialect(result.ids, "egy"))
# Skip token strings & offsets for max throughput
ids = msa_tok.encode_ids_with_dialect("مش عارف ايه اللي بيحصل", "egy")
texts = ["مش عارف ايه اللي بيحصل", "ده كلام فاضي"]
results = msa_tok.encode_batch_ids_with_dialect(texts, "egy")
| Attribute | Value |
|---|---|
| Base vocabulary | Nomeda-MSA-64Kv1 (64,000 tokens) |
| Plugin tokens added | 8,257 (16,000 target, filtered for uniqueness) |
| Combined vocabulary | 72,257 tokens |
| Plugin ID range | 64,000 – 72,256 |
| ID conflicts with base | 0 |
| Token conflicts with base | 0 |
| Training corpus | ~2B characters / 1M sentences sampled |
| Algorithm | SentencePiece Unigram (vocab_size=21,000, then filtered) |
| Test | Result | Source |
|---|---|---|
| MSA consistency | 20/20 (100%) | validation_e2e.json |
| Egyptian dialect word coverage | 41/41 (100%) | validation_e2e.json |
| Round-trip decode | 5/5 passed | validation_e2e.json |
| Word | MSA Base | Egyptian Plugin |
|---|---|---|
دلوقتي | 3 tokens | 1 token |
مبارح | 2 tokens | 1 token |
معايا | 2 tokens | 1 token |
معاه | 2 tokens | 1 token |
كلنا | 2 tokens | 1 token |
Measured on SaiedAlshahrani/Egyptian_Arabic_Wikipedia_20230101 — 10,000 text sample (~942K words, ~5.8M characters):
| Tokenizer | Fertility (↓) | Chars/Token (↑) | Throughput (chars/sec) |
|---|---|---|---|
| Nomeda MSA + EGY | 1.268 | 4.83 | 3,298,656 |
| MarBERT v2 | 1.317 | 4.65 | 1,466,027 |
| Nomeda MSA | 1.337 | 4.58 | 3,482,369 |
| AraBERT v02 | 1.470 | 4.17 | 1,685,620 |
| CAMeL Mix | 1.520 | 4.03 | 1,979,069 |
| GPT-4 (cl100k) | 4.270 | 1.44 | 5,389,809 |
Key findings:
encode_ids) in release buildDataset note: Egyptian Arabic Wikipedia is a mix of MSA and Egyptian dialect. The plugin's advantage is larger on pure dialect text.
┌─────────────────────────────────────────┐
│ Nomeda Tokenizer │
├─────────────────────────────────────────┤
│ Base Vocabulary (64K) │ Egyptian │
│ • MSA tokens │ Plugin (8K) │
│ • Shared across dialects│ • dialect │
│ • IDs 0–63999 │ tokens │
│ │ • IDs │
│ │ 64000+ │
└─────────────────────────────────────────┘
load_plugin(path, "egy").If you use this Egyptian plugin in research, please cite both the base tokenizer and this plugin repository.