afribert-kenya-adapted

license: apache-2.0

AfriBERT Kenya — Domain-Adapted Language Model

Rogendo/afribert-kenya-adapted is a continued pre-training of castorini/afriberta_large on a Kenyan language corpus using Masked Language Modeling (MLM).

It is optimised for Kenyan text: formal Swahili, Nairobi Sheng, M-PESA financial language, CPIMS child-protection terminology, and English-Swahili code-switching as used in everyday Kenyan communication.

---

What is Domain-Adaptive Pre-Training (DAPT)?

Standard AfriBERT was trained on African newswire and Wikipedia. While it understands Swahili well, it has never seen:

• Sheng slang (msee, poa, si poa, sawa kabisa)
• M-PESA vocabulary (Fuliza, Lipa na M-PESA, float, till number)
• CPIMS child-protection terminology (ustawi wa jamii, OVC, case worker, safe house)
• Kenyan WhatsApp code-switching patterns

DAPT is an intermediate training step between the base pretrained model and task-specific fine-tuning. It continues MLM pre-training on domain text so the model builds a richer internal representation of these patterns before learning any downstream task.

---

Training Data

The model was trained on five complementary sources totalling approximately 39 million tokens:

Note: CC-100 Swahili (uonlp/CulturaX) was available but disabled in the final run; sources 3 and 4 were repeated at high frequency so the model sees Kenyan domain text proportionally more than generic Wikipedia.

Synthetic Corpus (Source 3)

master_mlm_corpus.txt is a hand-crafted synthetic corpus covering:

• M-PESA transactions — sending, receiving, Fuliza overdraft, Lipa na M-PESA, Buy Goods
• CPIMS case language — intake forms, referrals, OVC (Orphans and Vulnerable Children), safe-house placements, court orders
• Sheng vocabulary — Nairobi urban slang integrated into Swahili sentences
• English-Swahili code-switching — meeting minutes, office messages, WhatsApp style

WhatsApp CPIMS Chat (Source 4)

Real WhatsApp export from a CPIMS field support group (whatsappchat-Bungoma.txt). Messages were filtered to remove media attachments and very short messages (<20 characters). This source was up-sampled ×20 because it contains the highest-quality real-world signal for the target domain despite its small size.

---

Training Configuration

---

Results

All evaluations were run on CPU using castorini/afriberta_large (base) vs Rogendo/afribert-kenya-adapted (adapted). Pseudo-perplexity is computed via sequential token masking — each token in the sentence is masked one at a time and the model's log-probability for the correct token is accumulated.

MLM Perplexity (lower = better)

Final MLM training perplexity: 5.39 (3 epochs, evaluated on 5% held-out set)

The two sentences where the adapted model is marginally worse (M-PESA and financial savings) both contain very common, unambiguous Swahili — the base model already predicts them near-perfectly. The largest gains are exactly where expected: Sheng (−66%) and English-Swahili code-switching (−41%).

Masked Token Prediction

Top-5 predictions per test, comparing base vs adapted model:

[Standard Swahili — Wikipedia style] Akiolojia ni somo linalohusu mabaki ya [tamaduni] za watu wa nyakati zilizopita.

Both models agree on tabia; adapted is more confident (0.363 vs 0.175).

---

[East African news — formal] Serikali imetangaza mpango mpya wa kukuza [uchumi] wa taifa kupitia biashara ya kimataifa.

Both models nail the correct answer with very high confidence — standard formal Swahili is well-represented in both.

---

[M-PESA domain — financial] Tuma [pesa] kwa kutumia nambari ya simu kupitia huduma ya M-PESA.

Adapted model places pesa in top-3; base model puts ujumbe (message) first — showing it doesn't understand M-PESA transaction context.

---

[CPIMS domain — child protection] Mtoto aliripotiwa kwa ofisi ya [ustawi] wa jamii baada ya kudhulumiwa nyumbani.

Both models strongly predict ustawi — child welfare language appears in Wikipedia. Adapted model is slightly more confident.

---

[Sheng / code-switching — Nairobi urban] Msee alikuwa poa sana, akanisaidia kupata [kazi] ya ofisi.

Base model puts kazi at rank 5 (3.7%). Adapted model's top predictions (pesa, emergency) reflect the CPIMS domain context — the model has learned that msee in an urban/office context relates to financial/emergency help.

---

[WhatsApp CPIMS — field worker message] Mtoto huyu ana umri wa miaka kumi na mbili na anahitaji [msaada] wa haraka.

Both models strongly predict msaada (help/assistance). Adapted model is significantly more confident (0.892 vs 0.774) — it has seen this exact phrasing repeatedly in WhatsApp CPIMS data.

---

[English-Swahili code-switch] Tulifanya meeting jana na manager akasema [project] itakuwa ready wiki ijayo.

The clearest demonstration of domain shift. Base model interprets meeting + manager as football context (Arsenal, league, club). Adapted model correctly understands it as an office/work context — system, team, process, salary are all semantically appropriate English loanwords.

---

Downstream Use — CPIMS Multi-Task Classifier

This model was used as the base encoder for Rogendo/cpims-nlp-intent-urgency, a multi-task classifier trained on 1,465 CPIMS support messages to predict:

• Intent (63 classes): login issues, password reset, data entry, escaped children, arrests, referrals, etc.
• Urgency (3 classes): high / medium / low

Results after full fine-tuning on the adapted base:

Compared to the previous version trained on distilbert-base-uncased with 271 rows: Intent F1 went from 46% → 74.5%.

---

Use Cases & Practical Domains

This model is designed for any NLP task involving Kenyan language text. It provides a stronger starting point than a generic multilingual model wherever the input contains Swahili, Sheng, code-switching, or Kenyan institutional vocabulary.

1. Child Protection & Social Work (CPIMS)

The primary motivation for this model. Kenya's Child Protection Information Management System (CPIMS) generates a high volume of support requests, case notes, and field reports written by social workers, case managers, and NGO staff — often in a mix of English, Swahili, and Sheng.

Practical tasks:

---

2. Financial Services & M-PESA

M-PESA is Kenya's dominant mobile money platform used by over 30 million Kenyans. Customer support queries, fraud reports, and transaction disputes are frequently written in Swahili or code-switched language that generic models mishandle.

Practical tasks:

---

3. Healthcare & Community Health Workers (CHWs)

Community Health Workers in Kenya file visit reports and referral notes, often verbally transcribed or typed on low-end phones in mixed Swahili/English.

Practical tasks:

---

4. Education & EdTech

Kenya's education sector uses a blend of English instruction and Swahili explanation, especially in lower grades. Many EdTech platforms serving rural Kenya receive student questions in Sheng or code-switched text.

Practical tasks:

---

5. Government & Civic Services

Kenya's e-citizen platforms, county service desks, and public feedback systems receive queries and complaints in everyday Kenyan language.

Practical tasks:

---

6. Media, Social Listening & Misinformation

Twitter/X, Facebook, and WhatsApp in Kenya carry a large volume of Kenyan Sheng and code-switched content that standard multilingual models struggle to classify.

Practical tasks:

---

Fine-tuning Guide

This model can be fine-tuned with as few as 200–500 labelled examples per class for most classification tasks, because DAPT has already adapted the internal representations to the target domain.

Recommended fine-tuning tasks by architecture

Minimum data guidelines

These estimates are based on domain-adapted models. A generic multilingual base model would need 3–5× more data to reach equivalent performance on Kenyan text.

Fine-tuning with HuggingFace Trainer

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer,
)

model_name = "Rogendo/afribert-kenya-adapted"
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=3  # e.g. urgency: low / medium / high
)

training_args = TrainingArguments(
    output_dir          = "my-kenya-classifier",
    num_train_epochs    = 5,
    per_device_train_batch_size = 16,
    learning_rate       = 2e-5,       # standard fine-tuning LR
    warmup_ratio        = 0.1,
    evaluation_strategy = "epoch",
    save_strategy       = "epoch",
    load_best_model_at_end = True,
    bf16                = True,       # use bf16 on A100/A40/H100
)

trainer = Trainer(
    model           = model,
    args            = training_args,
    train_dataset   = train_dataset,
    eval_dataset    = eval_dataset,
    processing_class = tokenizer,
)
trainer.train()

Fine-tuning with jenga_ai SDK (multi-task)

# cpims_config.yaml
model:
  base_model: Rogendo/afribert-kenya-adapted
  max_seq_len: 128

tasks:
  - name: intent
    task_type: multi_class_classification
    num_labels: 63
    label_column: intent

  - name: urgency
    task_type: multi_class_classification
    num_labels: 3
    label_column: urgency

training:
  epochs: 5
  batch_size: 16
  learning_rate: 2.0e-5
  output_dir: results/cpims-v2

python -m jenga_ai train --config cpims_config.yaml

---

Usage

Single mask prediction

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

tokenizer = AutoTokenizer.from_pretrained("Rogendo/afribert-kenya-adapted")
model = AutoModelForMaskedLM.from_pretrained("Rogendo/afribert-kenya-adapted")

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

# Real Sheng sentence — single mask
results = fill_mask(f"Oya, twendeni zetu, kuna {tokenizer.mask_token} flani ameniudhi. Uyo msee aliiba doh zangu most.")
for r in results:
    print(f"{r['token_str']:<20} {r['score']:.3f}")

Multiple masks (one position at a time)

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

tokenizer = AutoTokenizer.from_pretrained("Rogendo/afribert-kenya-adapted")
model = AutoModelForMaskedLM.from_pretrained("Rogendo/afribert-kenya-adapted")

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

# Multiple [MASK] tokens — pipeline returns a list of lists, one per mask position
results = fill_mask(
    f"Oya, twendeni zetu, kuna {tokenizer.mask_token} flani ameniudhi. "
    f"Uyo msee ameniibia {tokenizer.mask_token} zangu mingi sana nikimpata "
    f"{tokenizer.mask_token} sana, hadi atawacha kunibeba ufala."
)

for mask_predictions in results:
    print("--- New Mask ---")
    for r in mask_predictions:
        print(f"{r['token_str']:<20} {r['score']:.3f}")

As a base model for fine-tuning (jenga_ai SDK)

# experiment_config.yaml
model:
  base_model: Rogendo/afribert-kenya-adapted
  max_seq_len: 128

tasks:
  - name: intent
    task_type: multi_class_classification
    num_labels: 63
  - name: urgency
    task_type: multi_class_classification
    num_labels: 3

---

Limitations

• Not suitable for formal Standard Swahili tasks alone — the up-sampling of Sheng and code-switched text slightly shifts the model away from pure encyclopedic Swahili. Use castorini/afriberta_large directly for tasks that only involve formal Swahili prose.
• Sheng is not standardised — spelling varies by writer; the model reflects the patterns in the training WhatsApp data which may not generalise to all Sheng dialects (Mombasa Sheng differs from Nairobi Sheng).
• Small WhatsApp corpus — source 4 (real CPIMS field chat) is only ~30K tokens before repetition. Up-sampling compensates but does not replace volume.
• Private model — the model is currently private on HuggingFace Hub. Access requires a token with read permission on the Rogendo organisation.

---

Citation

If you use this model, please cite the base model:

@inproceedings{ogueji-etal-2021-small,
  title     = {Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages},
  author    = {Ogueji, Kelechi and Zhu, Yuxin and Lin, Jimmy},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Representation Learning},
  year      = {2021},
}

---

Author

Rogendo — built as part of the JengaAI CPIMS NLP pipeline for Kenyan child-protection support systems.