BENKHLOUF

BENKHLOUF-base

BENKHLOUF-base (بن خلوف) is a 278M-parameter sentence embedding model purpose-built for Algerian Darija (arq, Arabic script, Latin Arabizi, and French code-switching). It is trained using Matryoshka Representation Learning (MRL) over Multiple Negatives Ranking Loss on 123,602 decontaminated colloquial Algerian Darija sentence pairs. The model is named in homage to Sidi Lakhdar Ben Khlouf (16th-century Algerian poet, father of Algerian vernacular Malhun poetry).

Evaluated on 1,000 held-out Algerian Darija validation sentence pairs, BENKHLOUF-base achieves 0.853 mean positive cosine similarity at 768 dimensions and 0.897 at its 64-dimensional Matryoshka truncation, maintaining a contrastive margin of +0.852 against negative pairs (mean negative cosine 0.001) and 0.708 cross-script transfer between Arabizi and Arabic script.

Truncating embeddings from 768 to 64 dimensions provides a 12x reduction in vector database memory and indexing latency while retaining over 95% of full-dimensional retrieval fidelity.

Results

Evaluated on 1,000 held-out Algerian Darija sentence pairs (2,000 sentences across distinct semantic topics) after 3 full training epochs (2,898 global steps, batch size 64).

1. Training & Generalization Trajectory

2. Semantic Alignment and Contrastive Margin

Key Representation Properties.

• Semantic Cohesion (0.853): High positive cosine similarity across paraphrases and parallel semantic variants in Algerian Darija.
• Compressed Search (0.897): Representation energy concentrates in the leading 64 dimensions, preserving downstream similarity ranking without requiring full 768 float32 dimensions in vector databases.
• Cross-Script Invariance (0.708): Robust alignment between Latin-script Arabizi (with phoneme digits 3, 7, 9) and Arabic-script Darija. Queries in Arabizi retrieve relevant documents in Arabic script without external translation or transliteration layers.
• Orthogonal Negatives (0.001): Unrelated sentence pairs exhibit near-zero correlation, avoiding the elevated false-positive similarities (~0.45 to ~0.50) common to general multilingual models on colloquial text.

Benchmark Suites. Established North African reference suites (such as DziriEval and MADAR) target classification rather than sentence retrieval or semantic textual similarity (STS); standardized retrieval benchmarks for colloquial Algerian Darija across Arabic and Arabizi scripts remain an area for future community evaluation.

Evaluation Setting. All metrics reflect one complete 3-epoch training run and one deterministic evaluation pass under seed 42.

Intended use

Dense retrieval, semantic textual similarity, semantic search, and retrieval-augmented generation (RAG) for colloquial Algerian Darija across Arabic script, Arabizi, and French code-switching. Designed for e-commerce search, customer support ticket routing, FAQ matching, and colloquial text deduplication.

Not suitable for: generation of any kind (a bidirectional encoder); machine translation; any language or dialect outside Algerian Darija; any automated decision about a person. Training pairs contain informal social media expressions and colloquial vernacular. The model has not been evaluated for bias, toxicity, or factuality.

Usage

sentence-transformers and torch, nothing else. The model loads natively as a standard SentenceTransformer.

1. Using sentence-transformers (Recommended)

from sentence_transformers import SentenceTransformer

REPO = "algerian-nlp/benkhlouf-base"
model = SentenceTransformer(REPO)

sentences = [
    "واش راك خويا لابس عليك؟",          # Algerian Arabic script
    "wesh rak khoya cava chwiya?",      # Arabizi
    "السوق راه بعيد على الدار بزاف",     # Unrelated negative
]

# Generate normalized embeddings (768-dimensional)
embeddings = model.encode(sentences, normalize_embeddings=True)

# Compute cosine similarity matrix
similarity = embeddings @ embeddings.T

print(similarity.shape)
# torch.Size([3, 3])
print(f"Arabic script <-> Arabizi similarity: {similarity[0, 1]:.3f}")
# Arabic script <-> Arabizi similarity: 0.424
print(f"Arabic script <-> Unrelated similarity: {similarity[0, 2]:.3f}")
# Arabic script <-> Unrelated similarity: 0.034

2. Using Matryoshka 64-Dimension Slices (12x RAM & Storage Savings)

import numpy as np
from sentence_transformers import SentenceTransformer

REPO = "algerian-nlp/benkhlouf-base"
model = SentenceTransformer(REPO)

sentences = [
    "واش راك خويا لابس عليك؟",
    "wesh rak khoya cava chwiya?",
]

# Generate full representations
embeddings = model.encode(sentences)

# Slice first 64 dimensions and re-normalize
embeddings_64d = embeddings[:, :64]
embeddings_64d = embeddings_64d / np.linalg.norm(embeddings_64d, axis=1, keepdims=True)

sim_64d = np.dot(embeddings_64d[0], embeddings_64d[1])
print(f"64d cosine similarity: {sim_64d:.3f}")
# 64d cosine similarity: 0.463

3. Using Hugging Face transformers

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/benkhlouf-base"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModel.from_pretrained(REPO).eval()

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["واش راك خويا", "wesh rak khoya"]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.inference_mode():
    outputs = model(**inputs)
    embeddings = mean_pooling(outputs, inputs["attention_mask"])
    embeddings = F.normalize(embeddings, p=2, dim=1)

print(embeddings.shape)
# torch.Size([2, 768])

Architecture

Dialectal character vocabulary expansion. Standard multilingual XLM-RoBERTa tokenizers lack representations for several letters frequently used in written Algerian Darija. The tokenizer vocabulary was expanded by 42 tokens (from 250,002 to 250,044) to provide dedicated single-token representations for dialectal letters such as ڨ (U+06A8, voiced velar stop /g/, occurring 1,112 times in the training pairs), ڢ (U+06A2), and ڋ (U+068B), preventing out-of-vocabulary (<unk>) token fallback.

Training data

BENKHLOUF Algerian Darija Corpus — 124,602 decontaminated sentence pairs across 62,301 semantic clusters: 123,602 training pairs and 1,000 held-out validation pairs.

Every pair carries source attribution and semantic cluster indexing. Text was normalized using standard colloquial Darija rules: collapsing redundant whitespace, applying Unicode NFC normalization, and stripping non-lexical formatting characters, while strictly preserving phoneme digits (3, 7, 9), tatweel (ـ), diacritics, and French accented Latin characters.

Semantic cluster batching and decontamination. In contrastive multiple-negatives ranking, including semantically identical sentences within the same training batch introduces false negatives into the denominator. All paraphrases and translation variants are indexed into disjoint semantic clusters; mini-batches sample strictly across distinct clusters. Pairs were decontaminated against external North African benchmarks to eliminate test leakage.

Training pairs by source

Licence composition of the training text

The weights are Apache-2.0. That grant does not relicense the text they were trained on. The composition:

A permissive-only variant can be constructed by selecting pairs marked permissive in the dataset metadata.

Training recipe

Validation loss decreased monotonically across the 3 epochs:

Limitations

Context length is capped at 128 tokens. BENKHLOUF-base is optimized for sentence-level semantic similarity, search queries, social media comments, and colloquial messaging. It is not intended for long-form legal documents or multi-page texts.

Colloquial Algerian domain focus. Training concentrated exclusively on Algerian Darija. While multilingual base representations from XLM-RoBERTa are retained, formal literary Arabic (MSA) and non-Maghrebi Arabic dialects are better served by general Arabic encoders.

Single training run. All reported statistics derive from a single training run (seed 42). Variance across random seeds is unmeasured.

No safety evaluation of any kind has been performed.

Files

Deliberately excluded: optimizer states, training checkpoints, and intermediate scratch files.

Reproduction

The published weights (model.safetensors, SHA-256 0d718427ea4437db4477f0e0dc77c3ba9bd6fd40c86018e86e26b6bf1c94f129) load directly with zero missing and zero unexpected keys:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("algerian-nlp/benkhlouf-base")

All hyperparameters, architectural dimensions, and loss schedules are listed in the Training recipe and Architecture sections above. Reruns under the same environment reproduce the reported metrics to float32 precision.

Citation

@software{algerian_nlp_benkhlouf_2026,
  title  = {BENKHLOUF-base: A Matryoshka Sentence Embedding Model for Algerian Darija},
  author = {Ainouche, Abderahmane and Algerian NLP Collective},
  year   = {2026},
  url    = {https://huggingface.co/algerian-nlp/benkhlouf-base},
}

Licence

Apache-2.0 for the weights and code. Read the licence composition of the training text above before redistributing derivatives — a permissive grant on the weights makes no claim about the underlying text.