kw5-109M-instruct

KW5 109M Instruct

A 109.5M-parameter Swahili instruction model, LoRA fine-tuned from kw5-109M. It writes fluent Swahili in answer form. It does not read passages, does not decline questions it cannot answer, and gets facts wrong often. The measurements are below.

The newer kw5-149M-instruct-r2 is better at reading and at declining.

Built by Regnant.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-109M-instruct")
tok = AutoTokenizer.from_pretrained("regnant-io/kw5-109M-instruct")

messages = [{"role": "user", "content": "Orodhesha majina ya miji mitano ya Tanzania."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# add_special_tokens=False: the template already emits <s>.
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Output with torch.manual_seed(0) on CPU:

1. Dar es salaam
2. Dodoma
3. Morogoro
4. Tanga
5. Kilimanjaro
6. Arusha
7. Moshi
8. Zanzibar
9. Pwani
10. Jiji la Nairobi

Asked for five, it gave ten. Kilimanjaro and Pwani are regions, and Nairobi is in Kenya.

The shipped generation_config.json samples at temperature 0.2, top-p 0.9, repetition penalty 1.3.

Prompt format

<s>[INST] {user} [/INST] {assistant}</s>[INST] {user2} [/INST] {assistant2}</s>

with an optional system block inside the first [INST] only:

<s>[INST] <<SYS>>
{system}
<</SYS>>

{user} [/INST]

apply_chat_template produces exactly this. If you build prompts by hand:

1. Tokenize with add_special_tokens=False. The template already emits <s>.
2. Do not use <|system|>, <|user|> or <|assistant|> (ids 4 to 6). They never appeared in pretraining and LoRA left embed_tokens frozen, so those rows are untrained and prompting with them returns noise.

Keep prompt plus reply under 1024 tokens. The base model was trained at 1024 and its loss rises sharply past that point (see the base card).

Evaluation

The same 924-item Swahili suite is run on every KW5 instruct model, greedy, in each model's own chat format. Percent correct:

Passages come from TyDiQA-GoldP (Swahili dev) and Swahili Wikipedia; a reading answer counts if it contains the gold span or reaches token F1 0.5. The reading prompts are short (median 146 tokens), so the low score is not a context-length effect: the model answers from memory and ignores the passage.

Model details

Fine-tuning

The rank and module list were recovered from the published weights: the singular values of (merged minus base) cut off after 16 for every adapted projection, and every other tensor matches the base bit for bit.

The data combines regnant-io/swahili-instruction-22k (Apache 2.0) and Benjamin-png/swahili-instruction-mix, which is published under CC BY-NC 4.0. Only 137 of the 107,150 examples teach the model to say it does not know, which is consistent with the 2.5% above.

Limitations

• Does not use a supplied passage; see the evaluation.
• Almost never declines, so it answers unanswerable questions with inventions.
• Facts are unreliable. Do not use for factual lookup, medical, legal or financial advice.
• Count adherence is weak, as the quick start shows.
• Mostly Tanzanian Swahili. Minimal safety tuning. Not for unsupervised use.

Citation

@misc{kw5_109m_instruct_2026,
  title  = {KW5 109M Instruct: a Swahili instruction model},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-109M-instruct}
}

Apache 2.0 for the weights; see the note on training data licences above.