Downloads · 30 days
11
25% of all-time downloads
Karsamanja/swahili-english-whisper-cs-lora
swahili-english-whisper-cs-lora is a automatic speech recognition model from Karsamanja. Use it when you need speech turned into text. It is set up for peft. The card lists the license as cc-by-nc-4.0.
A LoRA adapter for openai/whisper-small fine-tuned for Swahili–English code-switched speech recognition (including Sheng slang), part of a bilingual voice-agent project for East Africa. The adapter keeps Whisper's mul…
Downloads · 30 days
11
25% of all-time downloads
All-time downloads
44
Public
Repo size
28.4 MB
Likes
1
Public
Click a slice to open those files.
.safetensors14.2 MB · 78%
From the Hugging Face model README
A LoRA adapter for openai/whisper-small fine-tuned for Swahili–English code-switched speech recognition (including Sheng slang), part of a bilingual voice-agent project for East Africa. The adapter keeps Whisper's multilingual weights frozen and adds low-rank adapters to the attention q_proj/v_proj layers, then is merged back into the base weights for deployment.
⚠️ Status: experimental quick-test checkpoint. This was trained with the notebook's
QUICK_TESTsetting (50 steps), not the full 5,000-step production run. It is intended to validate the training pipeline and is not a converged, production-quality model.
sw), English (en), and code-switched Swahili–English (incl. Sheng)liva-ai/swahili-english-asr) is CC-BY-NC. The base model (openai/whisper-small, MIT) does not impose this; retraining on CC0/own data would lift the non-commercial restriction.openai/whisper-small (244M params)Transcription of audio containing Swahili, English, or code-switched Swahili–English speech (e.g., conversational recordings, phone calls). Because the adapter was fine-tuned on code-switched data, it is intended to handle language mixing within a single utterance better than forcing a single language.
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
import torch
base_model_id = "openai/whisper-small"
adapter_id = "Karsamanja/swahili-english-whisper-cs-lora"
processor = WhisperProcessor.from_pretrained(adapter_id)
model = WhisperForConditionalGeneration.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)
# Optional: merge for deployment (no peft dependency at inference)
# model = model.merge_and_unload()
audio_array = ... # 16 kHz mono float32 numpy array
inputs = processor.feature_extractor(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
generated = model.generate(
inputs.input_features,
language=None, # auto-detect: do not force a language (code-switch mode)
task="transcribe",
temperature=0.0,
)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])
Designed to be dropped into a voice pipeline (e.g., faster-whisper/CTranslate2 after conversion, or the merged model) as the ASR component of a bilingual voice agent, where the language mix is detected per utterance and the LLM/TTS mirror the detected mix.
QUICK_TEST = False (5,000 steps) before any real use.liva-ai/swahili-english-asr; dialectal/Sheng coverage is limited and other East-African language varieties (Kiswahili sanifu vs. colloquial) may be underrepresented.Users should treat the reported metrics as pipeline artifacts, not quality claims, and should re-evaluate with a clean single-utterance benchmark before drawing conclusions about model quality.
Use the code below to get started with the model.
# Adapter-style loading (see Direct Use for the full snippet)
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(model, "Karsamanja/swahili-english-whisper-cs-lora")
processor = WhisperProcessor.from_pretrained("Karsamanja/swahili-english-whisper-cs-lora")
liva-ai/swahili-english-asr (~50h, conversational Swahili–English, CC-BY-NC, gated).Shelton1013/SwitchLingua_audio was loaded as an intended supplementary corpus in the training notebook but was not actually consumed by the quick-test training run.PEFT LoRA fine-tuning on frozen whisper-small. Encoder–decoder compatibility required a small PEFT wrapper that strips the leaked input_ids kwarg. Audio resampled to 16 kHz mono; text tokenized with max_length=448.
datasets Audio(sampling_rate=16000).input_ids (max 448); collator pads features and masks labels with -100.q_proj, v_proj]15 samples from liva-ai/swahili-english-asr (streaming, first 15), evaluated on CPU at batch size 2, language auto-detect. Indicative only — tiny, non-representative sample.
WER, CER, approximate per-language token WER (heuristic language detection), and PIER@3 (switch-window error rate).
| Metric | Fine-tuned (merged LoRA) | whisper-small baseline | whisper-tiny baseline (older) |
|---|---|---|---|
| Overall WER | 0.996 | 0.999 | 0.9996 |
| Overall CER | 0.962 | 0.968 | 0.995 |
| Swahili WER (approx.) | 0.998 | 1.000 | 1.000 |
| English WER (approx.) | 1.000 | 0.993 | 1.000 |
| PIER@3 | 0.998 | 0.996 | 1.000 |
These numbers are methodology artifacts, NOT quality measurements. The dataset references are full multi-turn speaker transcripts containing markers the model can never produce — e.g. Speaker 2 [00:00:00.000 - 00:00:04.930]: …, [laughing], [unintelligible] — while the model transcribes a single audio segment (≤30s window). Consequently every model saturates at WER ≈ 1.0 (whisper-tiny also scored 0.9996; one smoke run even produced WER > 1.0, which is structurally impossible with sane reference/prediction pairs). The fine-tuned model slightly edges the same-base baseline (0.996 vs 0.999 WER), but no conclusion about quality should be drawn. A fair evaluation requires a clean single-utterance benchmark (e.g., Common Voice sw, FLEURS sw, or turn-segmented transcripts).
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
Standard Whisper encoder–decoder (whisper-small) with LoRA adapters on attention projections, fine-tuned with the standard cross-entropy ASR objective.
Google Colab (T4 GPU / TPU v2-8 per notebook).
PyTorch, Transformers 5.x, PEFT (peft ≥ 0.8), datasets, evaluate/jiwer.
BibTeX:
@misc{karsamanja2026swahilienglishwhispercs,
title={Swahili--English Code-Switching Whisper (LoRA)},
author={Karsamanja},
year={2026},
howpublished={\url{https://huggingface.co/Karsamanja/swahili-english-whisper-cs-lora}},
note={Quick-test LoRA adapter fine-tuned from openai/whisper-small}
}
APA: Karsamanja. (2026). Swahili–English Code-Switching Whisper (LoRA) [Model]. Hugging Face. https://huggingface.co/Karsamanja/swahili-english-whisper-cs-lora
Karsamanja via Hugging Face