Emhotob-500K-MSA-Egyptian-v1

Emhotob-500K-MSA-Egyptian-v1 — Bidirectional MSA ↔ Egyptian Arabic (~0.5M params)

A 518K-parameter model finetuned to translate both ways between Modern Standard Arabic (الفصحى) and Egyptian colloquial Arabic (المصرية العامية), using a direction-specific Arabic system prompt to select the direction.

Finetuned from oddadmix/Emhotob-500K, a tiny Llama-architecture base (hidden size 16, 2 layers, 2 heads, tied embeddings).

⚠️ This is a scaling-study / capacity-limit reproduction, not a usable translator. It runs the exact SFT + evaluation recipe of the oddadmix/50M-MSA-Egyptian-v1 sibling on a base ~100× smaller (518K vs 51.8M params). At this size the model has far too little capacity for the task and collapses to a degenerate output (it emits a single repeated token), so BLEU/chrF are effectively zero. The purpose is to document how the same pipeline behaves as the base model shrinks — see the 50M sibling for a model that actually translates.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

The best checkpoint by validation loss is epoch 1 of 3 (eval_loss ≈ 8.42); validation loss does not improve after epoch 1 and the training loss plateaus around 8.44. For reference, uniform-random over the 32K vocab is ln(32000) ≈ 10.4 nats — so the model learns only the token-frequency prior, not the mapping. Greedy decoding of the resulting weights produces a degenerate, repeated-token string for essentially every input; the 20 saved samples per direction in eval_bidirectional.json show this directly.

What this is for

• A controlled scaling data point: identical data, prompts, tokenizer, ChatML masking, optimizer, schedule, and eval as the 50M run — only the base model capacity changes.
• A reproducible floor for the Emhotob translation suite.

If you want working MSA↔Egyptian translation, use oddadmix/50M-MSA-Egyptian-v1 (BLEU ~24–26 on the same held-out set).

Usage

Same ChatML interface as the rest of the suite (outputs will be degenerate at this size):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-500K-MSA-Egyptian-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EGY = "أنت مترجم محترف. ترجم النص من اللغة العربية الفصحى إلى اللهجة المصرية العامية."
SYS_TO_MSA = "أنت مترجم محترف. ترجم النص من اللهجة المصرية العامية إلى اللغة العربية الفصحى."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("شكرًا جزيلًا لك، أنت لطيف للغاية.", SYS_TO_EGY))

Training

• Base model: oddadmix/Emhotob-500K (Llama arch, hidden 16, 2 layers, 2 heads, vocab 32000, tied embeddings; 518,256 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations (132K rows, egyptian/msa columns)
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 129,009 train / 3,000 deterministic held-out (seed=42), scored both directions.

License

Apache-2.0, inherited from the base model.