Emhotob-5M-Darija-English-v1

Emhotob-5M-Darija-English-v1 — Bidirectional Darija ↔ English (~5M params)

A 5.08M-parameter model that translates both ways between Moroccan Darija (الدارجة المغربية) and English. A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M, a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

Scaling study. This runs the exact recipe of oddadmix/50M-Darija-English-v1 on a base ~10× smaller. At 5M it is one of the strongest of the 5M translation set (BLEU ~11–12 both ways), producing correct-language output with real lexical overlap, though it still drifts on longer inputs. A scaling demonstration, not a production translator; use the 50M sibling for fluent output.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

The saved weights are the best checkpoint by validation loss (eval_loss = 3.218, epoch 3 of 3). For reference, the 50M sibling scores BLEU ~40 (dar→en) / ~43 (en→dar) with the same data and eval.

Example translations

Real greedy-decoded outputs from the held-out set (rough — this is a 5M model):

Darija → English

English → Darija

The model reliably produces the right language and authentic Darija markers (غادي, كانتسنّا), but frequently loses source content on longer or noisier inputs. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-Darija-English-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EN  = "You are a professional translator. Translate the Moroccan Darija text into English."
SYS_TO_DAR = "أنت مترجم محترف. ترجم النص الإنجليزي إلى الدارجة المغربية."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("No, we have tickets darling", SYS_TO_DAR))

Training

• Base model: oddadmix/Emhotob-5M (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/darija_english_msa_parallel_dataset (90,104 rows; this model uses the darija and english columns). Sources: DoDA, HANTIFARAH combined, and ArabML/Skiredj parallel data.
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 87,104 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• A 5M model near the emergence threshold: correct language but frequent meaning loss, drift, and repetition on longer inputs.
• Darija has no standard orthography and mixes Arabic and Latin (Arabizi) script plus French/Amazigh loanwords — outputs vary in spelling and may hallucinate on out-of-domain input.
• For fluent translation use oddadmix/50M-Darija-English-v1.

License

Apache-2.0 (model weights, inherited from the base model). The training dataset aggregates several community corpora — check their individual licenses for downstream use.