Emhotob-5M-English-MSA-v2

Emhotob-5M-English-MSA-v1 — Bidirectional English ↔ MSA (~5.1M params)

A 5.1M-parameter model that translates both ways between English and Modern Standard Arabic (الفصحى). A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M-v2, a tiny Llama-architecture base (hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings).

Scaling study. This is one rung of a from-scratch Arabic scaling study that runs an identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where translation emerges. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M, becomes real-but-rough at 5M, and usable at 10M+. See the sibling oddadmix/50M-English-MSA-v1 for the fluent reference.

Evaluation

Deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Saved weights are the best checkpoint by validation loss (eval_loss = 1.802). 20 samples per direction with references are in eval_bidirectional.json.

Example translations

Real greedy-decoded outputs from the held-out set:

English → MSA

MSA → English

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-English-MSA-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."

def translate(text, system=SYSTEM):
    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

Training

• Base model: oddadmix/Emhotob-5M-v2 (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction).
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Eval split: 3,000 deterministic held-out pairs (seed=42), scored both directions.

Limitations

A ~5.1M model: reliable on short/common sentences, but drift, repetition, and errors appear on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the 50M sibling.

License

Apache-2.0, inherited from the base model.