Llama-3.2-1B-Kenyan-Languages-LoRA

Llama-3.2-1B-Kenyan-Languages-LoRA 🇰🇪✨

Llama-3.2-1B-Kenyan-Languages-LoRA is a lightweight parameter-efficient fine-tuned (PEFT / LoRA) instruction model built on top of unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit. It is tailored specifically for cross-lingual understanding, reasoning, and bidirectional translation across major low- and mid-resource Kenyan languages alongside English.

---

🌟 Key Features

• Supported Languages:
  • English (eng_Latn)
  • Swahili (swh_Latn)
  • Kikuyu / Gĩkũyũ (kik_Latn)
  • Luo / Dholuo (luo_Latn)
  • Kamba / Kikamba (kam_Latn)
• Direct Local-to-Local Capabilities: Trained not only on English-pivot translations, but also directly on indigenous inter-language pairs (e.g., Kikuyu ↔ Luo, Kamba ↔ Kikuyu, Luo ↔ Kamba).
• Strict Evaluation Decontamination: Rigorously decontaminated against FLORES+ dev and devtest evaluation benchmarks to guarantee zero data leakage during zero-shot and few-shot evaluation.
• Efficient Edge Deployment: 16-bit LoRA adapter weights (~100–300 MB) designed for rapid inference on constrained hardware, consumer GPUs, and edge runtimes.

---

📊 Tokenizer Fertility Metrics

Due to the subword vocabulary distribution of standard Llama tokenizers, sub-Saharan African languages encounter varying degrees of token fragmentation. Evaluated on the reference benchmark corpora:

Note: Given the higher fertility rates for Bantu languages like Kikuyu and Kamba, context windows (max_seq_length) should typically be set to 512–1024 tokens to avoid intermediate sequence truncation.

---

🚀 Quickstart & Inference

You can load and execute this model using Unsloth for 2x faster inference and minimal VRAM consumption:

from unsloth import FastLanguageModel
import torch

max_seq_length = 1024
dtype = None # Auto detection
load_in_4bit = True

# Load base model + LoRA adapters
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

FastLanguageModel.for_inference(model)

# Translation / Generation Prompt
prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert translator specializing in indigenous Kenyan languages.<|eot_id|><|start_header_id|>user<|end_header_id|>

Translate the following sentence into Kikuyu:
"Dr. Ehud Ur cautioned that the research is still in its early days."<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
response = tokenizer.batch_decode(outputs)
print(response[0].split("<|start_header_id|>assistant<|end_header_id|>")[-1].replace("<|eot_id|>", "").strip())





🛠️ Training Details
Trained by: Abel Kihara (theboulder123)
Base Architecture: Llama-3.2 1B Instruct (BNB 4-bit)
Optimization Framework: Unsloth & Hugging Face TRL (SFTTrainer)
PEFT Method: LoRA
LoRA Rank ($r$): 16
LoRA Alpha: 16
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Data Sources:
Filtered subsets of allenai/nllb (LASER score thresholding)
Silver pseudo-labeled bidirectional pairs with round-trip chrF verification
Complete decontamination against openlanguagedata/flores_plus

📜 Citation & Attribution
If you use this model or the associated local Kenyan language alignment weights in your research or application, please cite:
Code snippet
@misc{kihara2026llama_kenyan_languages,
  author = {Abel Kihara},
  title = {Llama-3.2-1B-Kenyan-Languages-LoRA: Parameter-Efficient Translation and Instruction Following for Kenyan Languages},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{[https://huggingface.co/theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA](https://huggingface.co/theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA)}}
}


⚖️ License
This project is open-sourced under the Apache-2.0 License. Compatible with the Meta Llama 3.2 Community License agreement.