MiniCPM5-1B-Twi

MiniCPM5-1B-Twi

A Twi (Akan) language adaptation of openbmb/MiniCPM5-1B, produced by tokenizer extension → continued pretraining (CPT) → supervised fine-tuning (SFT). The goal of this v1 is fluent Twi generation — it speaks natural, idiomatic Twi with correct Ghanaian orthography (ɛ/ɔ).

• Base model: openbmb/MiniCPM5-1B (a 1.1B-param Llama-architecture model)
• Languages: Twi/Akan (tw) primary; retains English
• Live demo: ghananlpcommunity/MiniCPM5-1B-Twi-Chat
• Training code & write-up: https://github.com/michsethowusu/minicpm5-twi-finetune

⚠️ Important: decoding settings

Use sampling with a repetition penalty. Under greedy decoding this model degenerates into repetition loops. Recommended:

temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3

These are set as defaults in generation_config.json.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "ghananlpcommunity/MiniCPM5-1B-Twi"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Kyerɛ me nkyerɛkyerɛmu tiawa fa Ghana aduane ho."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=True,
                     temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

Approach

1. Tokenizer extension. The base tokenizer split Twi very inefficiently — the Ghanaian vowels ɛ (U+025B) and ɔ (U+0254) each became two byte-level tokens, giving a Twi fertility of 2.90 tokens/word vs 1.29 for English. We trained an auxiliary byte-level BPE on clean Twi and added the top 6,000 Twi word-pieces to the tokenizer (130,560 → 136,560), then resized the model embeddings and warm-started each new token from the mean of its original subword embeddings. This cut Twi fertility to ~1.26 tokens/word (−57%).

2. Continued pretraining (CPT). Full-parameter CPT for 1 epoch (~666M tokens) on a mix of 65% monolingual Twi / 20% EN–Twi parallel / 15% English replay (the English replay mitigates catastrophic forgetting at this scale). Loss 6.29 → 1.90.

3. Supervised fine-tuning (SFT). LoRA SFT (rank 32) for 3 epochs on ~200k shuffled instruction/QA examples — a blend of short factual QA and longer passage-grounded QA in Twi. Loss 4.14 → 2.30.

Training data

All Twi text was Unicode-normalized to canonical Latin ɛ/ɔ before training.

Evaluation

Held-out Twi perplexity after CPT: 8.26. Qualitatively the model produces fluent, idiomatic Twi with correct orthography. Descriptive/explanatory prompts work well.

Limitations

This is a v1 focused on fluency, and it has real weaknesses:

• Factual accuracy is unreliable — output is fluent but can be wrong, and long answers sometimes drift or include invented words toward the end.
• Reasoning / math is weak — full-parameter CPT eroded some of the base model's instruction-following/reasoning ("alignment tax"), and the v1 SFT data was plain QA.
• Requires a repetition penalty (see above) — do not use greedy decoding.
• SFT data included machine-translated content, which introduces some translationese.

Planned v2 improvements: cleaner/curated SFT data, instruction-format English replay during CPT, and chain-of-thought-shaped examples (reason in English → answer in Twi).

Credits & license

Built on openbmb/MiniCPM5-1B (Apache-2.0) using datasets from the Ghana NLP community. Released under Apache-2.0. Developed by michsethowusu / Ghana NLP.