muzezuru

Muzezuru — Shona LoRA adapter for BLOOMZ-3B

A LoRA adapter fine-tuning bigscience/bloomz-3b toward fluent Shona conversation. Built on free-tier compute (Google Colab / Kaggle) with no budget, as an exploration of fine-tuning a capable base model on a low-resource language.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "bigscience/bloomz-3b"
adapter_repo = "REPLACE_WITH_YOUR_USERNAME/muzezuru"

tokenizer = AutoTokenizer.from_pretrained(adapter_repo)
base_model = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base_model, adapter_repo)

prompt = "### Instruction (Mirayiro):\nMhoro, wakadini?\n\n### Response (Mhinduro):\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3, no_repeat_ngram_size=3)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training details

• Base model: BLOOMZ-3B, chosen because Shona is explicitly included in its ROOTS pretraining corpus and xP3 instruction-tuning data.
• Method: QLoRA — 4-bit quantized base model, LoRA rank 16, alpha 32, applied to all linear layers (attention + feed-forward).
• Data: ~15,000 examples from a cleaned version of saillab/alpaca_shona_taco. The original dataset had a significant quality issue (84% of rows contained leftover "Instruction in English / Response in English" scaffolding from its generation pipeline) which was detected and fixed by extracting only the final Shona response before training.
• Compute: free-tier Kaggle Notebooks (T4 x2 GPU), ~2.5-3 hours.

Honest evaluation / known limitations

Evaluated across 15 prompts spanning greetings, factual questions, conceptual questions, instructions, code-switching, opinions, and casual conversation. Summary:

• Strongest on conceptual/philosophical questions (e.g. explaining ubuntu/unhu) — correctly uses native vocabulary and stays reasonably on-topic.
• Weak on factual/geographic questions — consistently hallucinates incorrect information (e.g. answering a question about Zimbabwe with facts about India or other countries). This appears to be a base-model knowledge gap rather than something fixable by more generic instruction fine-tuning.
• Weak on following concrete instructions (e.g. "write me a song" or "write me a letter") — often responds with unrelated content instead of performing the requested task.
• Weak on natural greetings/small talk — doesn't reliably respond to simple conversational openers.
• One notable strength: appropriately self-identifies as an AI lacking real-time information rather than fabricating an answer, in at least one tested case.

This is a research/portfolio project reflecting what is achievable fine-tuning a multi-billion-parameter model on a genuinely low-resource language using entirely free compute — not a production-ready assistant.

License

• Base model (BLOOMZ) is under the BigScience RAIL license — open, commercial use permitted, with a restriction against harmful use.
• Training data (alpaca_shona_taco) is CC BY-NC (non-commercial/research use only). This adapter is released for research/portfolio purposes accordingly; commercial use would require retraining on differently-licensed data.