50M-Darija-English-v1

50M-Darija-English-v1 — Bidirectional Darija ↔ English

A 51.8M-parameter small language model that translates both ways between Moroccan Darija (الدارجة المغربية) and English. A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/50M-2048-Emhotob, a tiny Arabic base model trained from scratch.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Both directions are genuinely hard: Darija has no single standardized orthography, is heavily borrowed/code-switched (French/Amazigh), and appears in the data in both Arabic script and Latin (Arabizi) script. The saved weights are the best checkpoint by validation loss (eval_loss=1.032, epoch 2 of 3).

Decoding note: use plain greedy. A repetition penalty (1.2) was tested across these 50M translation models and lowered BLEU by 5–12 points.

Example translations

Real greedy-decoded outputs from the held-out set:

Darija → English

(The second row shows the model handling Arabizi — Latin-script Darija — input.)

English → Darija

A larger set of 20 examples per direction (with references) is in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/50M-Darija-English-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EN  = "You are a professional translator. Translate the Moroccan Darija text into English."
SYS_TO_DAR = "أنت مترجم محترف. ترجم النص الإنجليزي إلى الدارجة المغربية."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya", SYS_TO_EN))
# → But it's my first flight and I'm a little scared
print(translate("I dare you to object to this topic", SYS_TO_DAR))
# → نتحداك تعارض هاد الموضوع

Training

• Base model: oddadmix/50M-2048-Emhotob (Llama arch, ~51.8M params)
• Dataset: oddadmix/darija_english_msa_parallel_dataset (90,104 rows; this model uses the darija and english columns). Sources: DoDA, HANTIFARAH combined, and ArabML/Skiredj parallel data.
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction, ~172.2K total). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 87,104 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• A 50M model: expect errors on rare / technical vocabulary, proper nouns, and long or noisy inputs. Everyday conversational text is handled best.
• Darija has no standard orthography and mixes Arabic and Latin (Arabizi) script plus French/Amazigh loanwords — outputs may vary in spelling and occasionally hallucinate on out-of-domain input. The training data also contains some crawled/wiki-formatting artifacts.
• Gender is disambiguated only from context; ambiguous inputs may default one way.
• For MSA or Egyptian pairs, see the sibling models oddadmix/50M-Darija-MSA-v1, oddadmix/50M-English-MSA-v1, and oddadmix/50M-Egyptian-English-v1.

License

Apache-2.0 (model weights, inherited from the base model). Note the training dataset aggregates several community corpora — check their individual licenses for downstream use.