twi-qwen2.5-0.5b-instruct

🇬🇭 Twi Fine-Tuned Language Model — twi-qwen2.5-0.5b-instruct

A Qwen2.5-0.5B-Instruct model fine-tuned on over 1.2 million Twi text examples for instruction-following and text generation in Twi (Akan), a language spoken by over 11 million people in Ghana.

📖 Blog: Building a Twi Language Model — From Data to Deployment

Why Twi?

Twi (also known as Akan Twi or Asante Twi) is one of the most widely spoken languages in Ghana, yet it remains severely underrepresented in modern NLP. Most large language models have little to no understanding of Twi, making them unreliable for Ghanaian users who want to interact in their native language.

This project aims to change that by fine-tuning a capable small language model specifically on Twi data, creating a foundation for Twi-language AI applications.

---

The Approach

1. Choosing the Base Model

We selected Qwen/Qwen2.5-0.5B-Instruct as our base model for several reasons:

• Multilingual foundation: Qwen2.5 was trained on diverse multilingual data, giving it a head start on understanding non-English scripts and character sets
• Small but capable: At 0.5B parameters, it's efficient to fine-tune and deploy, even on modest hardware
• Instruction-tuned: The instruct variant already understands conversational patterns, so we're adapting an existing capability rather than building from scratch
• Open license (Apache 2.0): Freely usable for any purpose

2. Curating the Twi Dataset

We combined two high-quality Twi datasets from the Ghana NLP Community:

Combined total: ~1,255,641 examples (1,192,858 train / 62,783 eval after a 95/5 split)

Each example was converted into a conversational format compatible with SFT training:

{
  "messages": [
    {"role": "user", "content": "Kyerɛ biribi fa Twi kasa ho."},
    {"role": "assistant", "content": "<twi text from dataset>"}
  ]
}

We filtered out examples shorter than 20 characters to ensure quality.

3. Training Configuration

We used TRL's SFTTrainer (Supervised Fine-Tuning) with the following configuration:

4. Key Implementation Details

Dataset Diversity: By combining narrative, dialogue, monologue, storyful, and news text, the model learns diverse Twi language patterns — from formal news reporting to casual conversation.

Memory Optimization: Gradient checkpointing and bf16 precision allowed us to train the full model (not LoRA) on a single A10G GPU while maintaining a reasonable batch size.

Evaluation: We held out 5% of the data for validation, with eval runs every 200 steps and checkpointing every 400 steps. The best checkpoint (lowest eval loss) is automatically selected.

---

How to Use the Model

Quick Start

from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="dicksonsarpong9/twi-qwen2.5-0.5b-instruct",
    device_map="auto"
)

messages = [{"role": "user", "content": "Kyerɛ me Ghana ho asɛm."}]
response = pipe(messages, max_new_tokens=512, temperature=0.7, do_sample=True)
print(response[0]["generated_text"][-1]["content"])

Detailed Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "dicksonsarpong9/twi-qwen2.5-0.5b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map="auto")

# Format as conversation
messages = [{"role": "user", "content": "Ka biribi fa aduane a Ghanafoɔ di ho."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    top_p=0.9,
    repetition_penalty=1.1,
)

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Interactive Chat (CLI)

Save the inference script and run:

# Install dependencies
pip install transformers torch accelerate

# Run examples
python inference_twi.py

# Single prompt
python inference_twi.py --prompt "Ɛdeɛn nti na adesua ho hia?"

# Interactive mode
python inference_twi.py --interactive

Example Prompts to Try

---

Training Data Sources & Acknowledgments

This model was made possible by the incredible work of the Ghana NLP Community, who have been building open-source NLP resources for Ghanaian languages. Their datasets represent a significant effort in documenting and digitizing Twi text across multiple domains.

• pristine-twi-english — A large-scale Twi dataset spanning four distinct writing styles (Narrative, Dialogue, Monologue, Storyful)
• twi-english-paragraph-dataset_news — Twi-English parallel paragraph dataset from news sources

Limitations

• Single-turn focus: The model was trained primarily on single-turn conversations. Multi-turn dialogue quality may be limited.
• Twi-only responses: The training focused on Twi text generation. The model may mix English and Twi in some responses, reflecting patterns in the training data.
• Knowledge cutoff: The model's knowledge is limited to what was in the base Qwen2.5 model plus the training data.
• Small model size: At 0.5B parameters, the model has limited reasoning capacity compared to larger models. It excels at fluent Twi text generation but may struggle with complex reasoning tasks.

Future Work

• Add more diverse Twi instruction data (Q&A, summarization, translation)
• Train larger variants (Qwen2.5-1.5B, 3B) for improved quality
• Add Twi-English translation capabilities
• Build evaluation benchmarks for Twi language understanding
• Create a Gradio demo Space for easy access

---

Technical Stack

License

This model is released under the Apache 2.0 license, following the base Qwen2.5 model license.

Citation

If you use this model, please cite:

@misc{twi-qwen2.5-0.5b-instruct,
  author = {Dickson Sarpong},
  title = {Twi Fine-Tuned Qwen2.5-0.5B-Instruct},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/dicksonsarpong9/twi-qwen2.5-0.5b-instruct}
}

---

Yɛn kasa yɛ kɛseɛ. Our language is great. 🇬🇭