kenyan-gpt

Kenyan Conversational Qwen 2.5

This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the micymike/kenyan-conversational-data dataset.

Model Description

• Developed by: Michael Moses & VilyaLabs
• Model type: Causal Language Model (Fine-tuned with LoRA)
• Language(s) (NLP): English, Swahili, Sheng (Kenyan context)
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen2.5-1.5B-Instruct

Training Details

• Hardware: Trained on TPU v5e
• Epochs: 1
• Batch Size: 4 (with Gradient Accumulation Steps: 4)
• Learning Rate: 3e-5
• Optimizer: AdamW
• Precision: bfloat16

LoRA Parameters

• R: 16
• Alpha: 32
• Dropout: 0.05
• Target Modules: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj

Intended Use

This model is designed to assist with conversational tasks within a Kenyan cultural and linguistic context, incorporating localized slang and references where appropriate.

How to Use

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_model_id = "VilyaLabs/kenyan-gpt" # or micymike/kenyan-gpt

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_model_id)