Emhotob-5M-Egyptian-English-v1

Emhotob-5M-Egyptian-English-v1 — Bidirectional English ↔ Egyptian Arabic (~5M params)

A 5.08M-parameter model that translates both ways between English and Egyptian colloquial Arabic (المصرية العامية). A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M, a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

Scaling study. This runs the exact recipe of oddadmix/50M-Egyptian-English-v1 on a base ~10× smaller. At 5M the model produces fluent-looking target-language text and switches language/register correctly, but the content is often wrong — it drifts and repeats heavily. Cross-lingual English↔Egyptian is measurably harder at this size than the intra-Arabic MSA↔Egyptian pair. It is a scaling demonstration, not a usable translator; use the 50M sibling for real translation.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

The saved weights are the best checkpoint by validation loss (eval_loss = 3.367, epoch 3 of 3). For reference, the 50M sibling scores BLEU ~24 (en→egy) / ~34 (egy→en) with the same data and eval.

Example translations

Real greedy-decoded outputs from the held-out set (rough — this is a 5M model):

English → Egyptian

Egyptian → English

The model emits well-formed Egyptian / English surface text and picks the right language for the direction, but frequently loses the source meaning. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-Egyptian-English-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EGY = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللهجة المصرية العامية."
SYS_TO_EN  = "You are a professional translator. Translate the Egyptian Arabic text into English."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("I will be fine, don't worry", SYS_TO_EGY))

Training

• Base model: oddadmix/Emhotob-5M (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-translation-dataset-2.9-openai-batch (135,282 rows, English/Arabic columns; Arabic = Egyptian colloquial)
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 132,282 train / 3,000 deterministic held-out (seed=42), scored both directions.

<!-- BEGIN: leaderboard-en2egy-eval -->

Out-of-domain evaluation — Egyptian Arabic Translation Benchmark

The results above are in-domain: a held-out split of the same corpus this model was trained on. The numbers below are out-of-domain — the same model scored on the Egyptian Arabic Translation Benchmark (oddadmix/egyptian-arabic-translation-benchmark, 319 English→Egyptian pairs written by a different annotator with different orthographic conventions).

Expect these to be substantially lower than the in-domain scores. That gap is the generalization penalty, not a regression — both numbers are real, they measure different things.

Decoding is deterministic greedy (do_sample=False, no repetition penalty), using the exact ChatML prompt format the model was trained with — the same protocol as every other number in this study.

Where this rung sits

Reading these numbers

BLEU understates quality on this set. Scoring is against a single reference, so a correct translation that picks a different valid word is penalized — e.g. فريش vs the reference's طازة for "fresh", or التليفون اللي ضاع vs تليفونها الضايع for "her lost phone". Both are good Egyptian; only one matches the reference. chrF and METEOR track perceived quality more closely here.

At 319 rows, differences of roughly 1–2 BLEU between adjacent rungs are within noise.

These are small models — 5M to 50M parameters, orders of magnitude below the large systems typically evaluated on this benchmark. The result of interest is the scaling curve and per-parameter efficiency, not absolute rank against models 100–1000× the size.

<!-- END: leaderboard-en2egy-eval -->

Limitations

• A 5M model near the emergence threshold: fluent surface text but frequent meaning loss, drift, and repetition — especially on longer or rarer inputs.
• Gender is disambiguated only from context; ambiguous inputs may default one way.
• For fluent translation use oddadmix/50M-Egyptian-English-v1.

License

Apache-2.0, inherited from the base model.