SedibaLM

SedibaLM V7

A Sepedi (Sesotho sa Leboa) language model, fine-tuned from Qwen2.5-1.5B-Instruct with QLoRA by Sediba AI NPC.

Sepedi is spoken by roughly 4.7 million people in South Africa and is severely under-represented in language models. V7 is the first validated model in the SedibaLM lineage.

---

⚠️ Read this first: you MUST use the chat template

V7 is a ChatML-format SFT model. Prompted with raw text it emits EOS immediately and returns an empty string. This is the single most common way to conclude "the model is broken" when it is not.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Sedibaai/SedibaLM"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)
model.eval()

# ✅ CORRECT — wrap the prompt in the chat template
messages = [{"role": "user", "content": "Ngwala ka bohlokwa bja thuto ka Sepedi."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=128,
        temperature=0.7,
        top_p=0.8,
        repetition_penalty=1.3,     # recommended; prevents degenerate loops
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

# decode ONLY the new tokens — do not slice by prompt string length
print(tokenizer.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))

# ❌ WRONG — returns "" (immediate EOS)
inputs = tokenizer("Ngwala ka bohlokwa bja thuto ka Sepedi.", return_tensors="pt")

Two further gotchas we hit ourselves:

• Decode the new-token tail (out[0][inputs.input_ids.shape[-1]:]). Slicing decoded text by len(prompt) breaks once the input is a chat-templated wrapper.
• On transformers 4.44 and older, chat_template.jinja is not auto-loaded — read the file and pass it to apply_chat_template(chat_template=...).

---

What V7 actually changes

Given Sepedi input, the base Qwen2.5-1.5B-Instruct refuses in English and misidentifies the language:

"I apologize for any misunderstanding but I don't have enough context to provide an accurate translation… The text appears to be written using the Sesotho sa Leboa language (also known as Setswana), which is spoken primarily in South Africa among people from Botswana's northern region…"

That is wrong on two counts: Sesotho sa Leboa is Sepedi, and it is not Setswana from northern Botswana.

V7, same input, answers in Sepedi:

Tumalanong ye ke gore re tla dira dilo ka mokgwa wa temothuo gomme ra fediša ditlamorago tše di sa kgahlišego kudu mo nageng ya gaborena.

The achievement is that V7 treats Sepedi as a language it speaks, rather than a foreign artefact to apologise about.

---

Evaluation

Held-out slice of 1,109 Sepedi-only records (filtered by function-word density), n=50, 80 max new tokens, identical prompts and harness for both runs.

SCORE = 0.5·chrF + 0.3·monolinguality + 0.2·no_repetition

How to read these numbers honestly

These are comparative figures for ranking checkpoints. They are not absolute quality claims, and we would rather say so than let them be over-read:

• Monolinguality 0.1934 does NOT mean "19% Sepedi." It is a deliberately conservative heuristic: a token counts as Sepedi only if it carries a Bantu diacritic (š ŋ ñ ê ô à ë ü) or matches a digraph list (kg ny tl dl sw rw ph kh tsh th sh ngw). Perfectly ordinary Sepedi words with neither feature score as ambiguous, not as Sepedi. Fully-Sepedi output still lands in the 0.15–0.19 band. Read the 2.99× gap, not the absolute value.
• chrF must be continuation-only. Scored against the whole record (prompt included), chrF actually inverts and favours the base model (0.1729 vs V7's 0.1492) — because the base model echoes the Sepedi prompt back inside its English apology, which scores n-gram overlap, while V7 writes fresh Sepedi. Excluding the prompt raises V7 +47.6% but the base only +8.1%. Any chrF number for this model that doesn't say "continuation-only" is measuring prompt echo.
• The task is open-ended continuation, not translation, so chrF structurally understates quality.

---

Specifications

Training loss fell 6.39 → 1.92.

Note on vocabulary. A 400-subword Sepedi tokenizer extension (vocab 152,065) was designed for this lineage but was not consumed by this run — V7 is stock 151,665. If you see 152,065 quoted anywhere, that is design intent for a future V8, not this model. Also note Qwen2.5's embedding matrix is padded to 151,936, which is larger than the vocab by design; do not call resize_token_embeddings to "fix" it.

---

Limitations

• Not aligned. Supervised fine-tuning only — no RLHF, no DPO. No safety tuning.
• Modest continuation chrF (0.22). V7 continues in the right language and register but does not reproduce reference text closely.
• Code-switching. V7 can still emit English or mixed output.
• Small base. 1.5B parameters; not comparable to frontier models on reasoning.
• No formal data-licence audit has been completed on the training corpus.
• Tokenizer inefficiency. Stock Qwen2.5 tokenizes Sepedi at ~2.09 tokens/word; the planned vocab extension projects ~1.82 (−12.6%).
• No formal benchmark suite. V7 is evaluated on an internal held-out slice, not on a public benchmark (e.g. MasakhaNER, AfriMMLU). Cross-model comparison is therefore not yet possible.

Intended use

Sepedi conversational AI; community information delivery; low-resource language modelling research; demonstrating sovereign African AI capability.

Out of scope

Medical, legal, or financial advice. Safety-critical or real-time systems. Languages other than Sepedi.

---

Licensing

Model weights: CC BY-SA 4.0. Attribution to Sediba AI NPC and contributing communities; derivatives share alike.

The training data is not released under this licence. Corpus material is governed separately by NOODL (Northern-Sotho Open Data Licence, in legal review) — the open-weights layer and the restricted-data layer are deliberately distinct.

Ethics and provenance

Community-contributed data was gathered under FPIC (Free, Prior and Informed Consent) principles. The Kutullo Exchange royalty mechanism is designed so contributing communities share in downstream value. Do not use this model to generate content that misleads or harms Sepedi-speaking communities.

Citation

@misc{sedibalm-v7-2026,
  title  = {SedibaLM V7: A Sepedi (Sesotho sa Leboa) Language Model Fine-Tuned with QLoRA},
  author = {Sediba AI NPC},
  year   = {2026},
  url    = {https://huggingface.co/Sedibaai/SedibaLM}
}

---

Built by Sediba AI NPC — sovereign AI for South African languages. Sepedi first, then the remaining official languages of South Africa, then the wider Global South.