kw5-109M

KW5 109M

A 109.5M-parameter Swahili (Kiswahili) base language model, pretrained from scratch on 1.41B tokens on a single NVIDIA T4. This is the first KW5 model. Its successor is kw5-149M.

Built by Regnant.

This is a base model. It does next-token prediction only. The instruction tuned version is kw5-109M-instruct.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-109M")
tok = AutoTokenizer.from_pretrained("regnant-io/kw5-109M")

# A base model continues a prefix; it does not follow instructions.
inputs = tok("Tanzania ni nchi", return_tensors="pt", add_special_tokens=False)
out = model.generate(**inputs, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

Output with torch.manual_seed(0) on CPU (sampling is on, so yours will differ):

Tanzania ni nchi ya Afrika Mashariki na Kati. Kwa hiyo, kwa kuwa tunapakana na
Kenya, Uganda, Rwanda, Burundi, DRC Kongo (DRC), Zambia, Malawi, Msumbiji,
Mauritius, Seychelles, Mozambique, Swaziland, Zimbabwe, Comoro, ...

Fluent, and wrong: Mauritius, Seychelles, Zimbabwe and Swaziland do not border Tanzania. That is typical of this model.

No <s> is needed. Pretraining packed documents without a leading BOS. An earlier revision of this card said every training sequence began with <s>; it did not. Adding one anyway is harmless: on held-out text it moves the loss by 0.009 nats.

The shipped generation_config.json samples at temperature 0.2, top-p 0.9, repetition penalty 1.3, which is what the output above used.

Model details

Context: use 1024 tokens

The config allows 2048 positions, but the model was only ever trained at 1024. The planned extension to 2048 was gated behind a decay phase that never ran (see Training), so rope_theta is still 10,000. Measured loss by position on 12 long Swahili documents:

Past 1024 tokens the model is reading positions it never saw. Keep prompt plus output under 1024.

Tokenizer

32k BPE trained from scratch on the deduplicated Swahili corpus. Special ids: <unk>=0, <s>=1, </s>=2, <pad>=3. Ids 4 to 6 (<|system|>, <|user|>, <|assistant|>) are reserved but never occur in the pretraining corpus, so their embedding rows are untrained. If you fine-tune with them as role markers, train embed_tokens too.

Training

The learning rate was never decayed. The token budget was set to 8B from a corpus-size estimate that turned out to be about four times too high, so the decay phase was scheduled for a point the run never reached. Training stopped at 1.41B tokens with the learning rate still at 3e-4. There was no held-out validation set in this run. An earlier revision of this card said the decay was skipped on purpose because validation quality was falling; that was not what happened.

Because packing had no document mask, the model was trained to attend across unrelated documents inside one sequence.

Evaluation

Loglikelihood multiple choice, n = 50 per task:

Both are consistent with chance. With 50 items this is a floor, not a capability. The model's one real skill is fluent Swahili continuation.

Intended use

A small starting point for Swahili experiments and fine-tuning. For new work, start from kw5-149M instead.

Not for deployment: no instruction following, no safety tuning, poor factual reliability.

Limitations

• It continues text; it does not answer questions.
• Facts are unreliable, as the sample above shows.
• Mostly Tanzanian Swahili web text, with the biases of web text.
• No safety tuning at all.

Files

Citation

@misc{kw5_109m_2026,
  title  = {KW5 109M: a Swahili base language model trained on a single T4},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-109M}
}

Apache 2.0.