asr-whisper-helpline-sw-v1

asr-whisper-helpline-sw-v1

This model is a fine-tuned version of openai/whisper-large-v2 on the Common Voice Swahili dataset.

Model Description

This ASR model is specifically fine-tuned for Swahili speech recognition in the context of the Tanzania Child Helpline, powered by OpenCHS (Open Source Child Helpline System). The model is designed to transcribe Swahili spoken in Tanzanian call center environments.

Performance Highlights:

• Best Validation WER: 23.56% (achieved at step 7,500 of continued training)
• Baseline WER: 89.05% (Whisper Large v2 zero-shot on Common Voice 17.0)
• Improvement: ~65.5 percentage point reduction in WER (~73.5% error rate reduction)

This represents a significant improvement over the base Whisper Large v2 model for Swahili transcription tasks.

Training Strategy

The model was trained in two stages:

1. Stage 1 - Common Voice 17.0: Initial fine-tuning on Common Voice 17.0 Swahili dataset (10,000 steps)
2. Stage 2 - Common Voice 23.0: Continued fine-tuning on Common Voice 23.0 Swahili dataset (7,500 steps)

Total Training: 17,500 effective steps with the best checkpoint selected at step 7,500 of stage 2 based on lowest validation WER.

Intended Uses & Limitations

Intended Uses

• Primary: Transcribing Swahili speech in call center environments, specifically for child helpline services in Tanzania
• General: Swahili automatic speech recognition tasks
• Research: Baseline for domain adaptation studies (general speech → telephony/call center audio)

Limitations

• Domain Shift: Model is trained on Common Voice (clean, read speech) but intended for call center audio. Performance on actual telephony audio may differ and requires validation.
• Language Variety: Training data may not fully represent all Tanzanian Swahili dialects and speaking styles.
• Audio Quality: Performance may degrade with low-quality audio, background noise, or poor recording conditions typical in telephony.
• Code-Switching: May not handle code-switching between Swahili and English/other languages well.

Known Issues

• Domain-specific evaluation on actual call center audio is pending

Training and Evaluation Data

Stage 1: Common Voice 17.0 (Swahili)

Training Configuration:

• Training samples: Streamed entire Common Voice 17.0 Swahili training split
• Validation samples: 2,000 samples
• Source: Mozilla Common Voice 17.0
• Language: Swahili (sw)
• Data type: Read speech from diverse speakers
• Streaming mode: Used dataset streaming to minimize disk usage

Stage 1 Results:

• Final validation WER: 23.62%
• Training steps: 10,000

Stage 2: Common Voice 23.0 (Swahili)

Training Configuration:

• Starting point: Best checkpoint from Stage 1
• Training samples: Common Voice 23.0 Swahili training split (downloaded locally)
• Validation samples: 2,000 samples
• Source: Mozilla Common Voice 23.0
• Language: Swahili (sw)

Stage 2 Results:

• Best validation WER: 23.56% at step 7,500
• Training continued to 10,000 steps but early stopping applied retrospectively

Baseline Performance:

• Base Whisper Large v2 (zero-shot): 89.05% WER on Common Voice 17.0 validation

Training Procedure

Training Hyperparameters - Stage 1 (Common Voice 17.0)

Optimization:

• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• optimizer: AdamW (torch) with betas=(0.9, 0.999) and epsilon=1e-08
• training_steps: 10,000

Batch Configuration:

• train_batch_size: 16
• eval_batch_size: 16
• gradient_accumulation_steps: 1

Memory Optimization:

• gradient_checkpointing: true
• mixed_precision_training: Native AMP (FP16)
• dataloader_num_workers: 2

Evaluation & Checkpointing:

• evaluation_strategy: steps (every 500 steps)
• save_steps: 500
• logging_steps: 50

Other:

• seed: 42

Training Hyperparameters - Stage 2 (Common Voice 23.0)

Optimization:

• learning_rate: 5e-06 (reduced from Stage 1)
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 500
• optimizer: AdamW (torch) with betas=(0.9, 0.999) and epsilon=1e-08
• training_steps: 20,000 (stopped at 10,000, best at 7,500)

Batch Configuration:

• train_batch_size: 16
• eval_batch_size: 16

Memory Optimization:

• mixed_precision_training: Native AMP (FP16)

Evaluation & Checkpointing:

• evaluation_strategy: steps (every 500 steps)
• save_steps: 500
• logging_steps: 50

Other:

• seed: 42

Training Results - Stage 2 (Common Voice 23.0)

Training Observations:

• Initial performance on CV23: 24.80% WER (step 500)
• Progressive improvement to best WER of 23.56% at step 7,500
• Performance degraded slightly after step 7,500 (overfitting indicators)
• Model weights restored to step 7,500 checkpoint for optimal performance

Combined Training Summary

Stage 1 (CV 17.0):

• Steps: 0 → 10,000
• Starting WER: 43.68% → Final WER: 23.62%

Stage 2 (CV 23.0):

• Steps: 0 → 7,500 (best checkpoint)
• Starting WER: 24.80% → Best WER: 23.56%

Total Effective Training: ~17,500 steps across two datasets

Performance Comparison

Note: The two-stage training approach with dataset progression (CV 17.0 → CV 23.0) achieved marginal improvement in final WER while ensuring model robustness across Common Voice versions.

Usage

from transformers import pipeline

# Load the model
pipe = pipeline("automatic-speech-recognition",
                model="openchs/asr-whisper-helpline-sw-v1")

# Transcribe audio
result = pipe("path/to/swahili_audio.wav")
print(result["text"])

Advanced Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

# Load model and processor
processor = WhisperProcessor.from_pretrained("openchs/asr-whisper-helpline-sw-v1")
model = WhisperForConditionalGeneration.from_pretrained("openchs/asr-whisper-helpline-sw-v1")

# Load and process audio
# ... your audio loading code ...

# Generate transcription
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription[0])

Future Work

• Domain Evaluation: Assessment on actual Tanzania Child Helpline call center audio to measure domain shift impact
• Domain Adaptation: Fine-tuning on telephony/call center audio for improved production performance
• Error Analysis: Detailed analysis of failure cases to identify improvement opportunities
• Test Set Evaluation: Comprehensive evaluation on Common Voice 23.0 test split

Citation

If you use this model, please cite:

@misc{openchs-swahili-asr-v1,
  title={Swahili ASR Model for Tanzania Child Helpline},
  author={OpenCHS Team},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/openchs/asr-whisper-helpline-sw-v1}}
}

Framework Versions

• Transformers: 4.56.2
• PyTorch: 2.8.0+cu128
• Datasets: 2.21.0
• Tokenizers: 0.22.1

License

Apache 2.0

Acknowledgments

• Base model: OpenAI Whisper Large v2
• Training data: Mozilla Common Voice 17.0 and Mozilla Common Voice 23.0
• Project: OpenCHS