naija-csm-finetuned-model

Naija CSM Fine-tuned Text-to-Speech Model

<!-- Provide a quick summary of what the model is/does. -->

A fine-tuned CSM (Conditional Speech Model) for generating natural-sounding Nigerian English speech from text. This model specializes in producing TTS output with authentic Nigerian accent and pronunciation patterns.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is a fine-tuned Text-to-Speech (TTS) model based on the CSM (Conditional Speech Model) architecture, specifically trained on Nigerian English audio data to generate natural-sounding speech with Nigerian accent and pronunciation patterns. The model uses LoRA (Low-Rank Adaptation) fine-tuning for efficient training and maintains the multi-speaker capabilities of the base model.

• Developed by: Sam4rano
• Model type: Text-to-Speech (TTS) / Conditional Speech Model
• Language(s) (NLP): English (Nigerian variant)
• License: Apache 2.0
• Finetuned from model: unsloth/csm-1b

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://huggingface.co/Sam4rano/naija-csm-finetuned-model
• Base Model: unsloth/csm-1b
• Training Framework: Unsloth

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

This model can be used to generate Nigerian-accented English speech from text input. It's particularly useful for:

• Creating TTS systems for Nigerian applications
• Generating natural-sounding Nigerian English audio for educational content
• Voice synthesis for Nigerian-focused applications and services

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

• Audiobook narration with Nigerian accent
• Educational tools for Nigerian English pronunciation

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

• This model is not suitable for generating speech in languages other than Nigerian English
• Should not be used for creating misleading, deceptive, or harmful audio content
• Not recommended for real-time applications without proper optimization
• Not suitable for other English variants or accents without additional fine-tuning

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Accent Specificity: Trained primarily on Nigerian English, may not perform well for other English variants
• Speaker Diversity: Quality may vary depending on the diversity of speakers in the training data
• Audio Quality: Output quality depends on the quality and diversity of the training dataset
• Computational Requirements: Requires GPU for optimal inference speed
• Limited Training: Model was trained for only 60 steps as a demonstration - production use may require additional training

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be aware that this model is optimized for Nigerian English and may not generalize well to other accents or languages. Consider additional training with more data and epochs for improved performance.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import CsmForConditionalGeneration, AutoProcessor
import torch
import soundfile as sf

# Load the model and processor
model = CsmForConditionalGeneration.from_pretrained("Sam4rano/naija-csm-finetuned-model")
processor = AutoProcessor.from_pretrained("Sam4rano/naija-csm-finetuned-model")

# Generate speech
text = "Welcome to Nigeria, the giant of Africa!"
speaker_id = 0

# Method 1: Simple text input
inputs = processor(f"[{speaker_id}]{text}", add_special_tokens=True)
audio_values = model.generate(
    **inputs,
    max_new_tokens=125,  # 125 tokens ≈ 10 seconds of audio
    output_audio=True
)

# Convert to audio array and save
audio = audio_values[0].to(torch.float32).cpu().numpy()
sf.write("nigerian_speech.wav", audio, 24000)

# Method 2: Conversation format (recommended)
conversation = [
    {"role": str(speaker_id), "content": [{"type": "text", "text": text}]},
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
)

audio_values = model.generate(
    **inputs,
    max_new_tokens=125,
    output_audio=True
)

audio = audio_values[0].to(torch.float32).cpu().numpy()
sf.write("nigerian_speech.wav", audio, 24000)

# For voice consistency, provide audio context
sample_audio = your_reference_audio_array  # numpy array at 24kHz
sample_text = "Your reference text"

conversation_with_context = [
    {"role": str(speaker_id), "content": [
        {"type": "text", "text": sample_text},
        {"type": "audio", "path": sample_audio}
    ]},
    {"role": str(speaker_id), "content": [{"type": "text", "text": text}]},
]

inputs = processor.apply_chat_template(conversation_with_context, tokenize=True, return_dict=True)
audio_values = model.generate(**inputs, max_new_tokens=125, output_audio=True)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was trained on a custom Nigerian English audio dataset containing text-audio pairs with speaker information. The dataset was preprocessed to ensure consistent formatting and quality.

Data characteristics:

• Language: Nigerian English
• Audio format: 24kHz sampling rate
• Speaker handling: Multi-speaker support with consistent speaker IDs
• Text processing: Tokenized with special speaker tokens

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing

The training data underwent several preprocessing steps:

• Audio files resampled to 24kHz sampling rate
• Maximum audio length: 240,001 samples (approximately 10 seconds)
• Maximum text length: 256 tokens
• Speaker IDs converted to consistent string format
• Audio padding to maximum length for batch processing
• Text padding with attention masks

Training Hyperparameters

• Training regime: Mixed precision (fp16/bf16) with LoRA fine-tuning
• Base model: unsloth/csm-1b
• LoRA Configuration:
  • LoRA rank (r): 32
  • LoRA alpha: 32
  • LoRA dropout: 0
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4
• Batch size: 2 (per device)
• Gradient accumulation steps: 4
• Training steps: 60
• Warmup steps: 5
• Optimizer: AdamW 8-bit
• Weight decay: 0.01
• LR scheduler: Linear
• Random seed: 3407

Speeds, Sizes, Times

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

• Training time: Approximately 2-5 minutes on Tesla T4 GPU
• Memory usage: ~8-12GB GPU memory during training
• Model size: LoRA adapters only (~few MB), requires base model for full functionality
• Hardware: Google Colab Tesla T4 GPU (16GB VRAM)

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

Due to the demonstration nature of this training (60 steps), formal evaluation was limited to basic functionality testing and audio generation quality assessment.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

• Audio quality and naturalness
• Speaker consistency
• Pronunciation accuracy for Nigerian English
• Text-to-speech alignment

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

• Qualitative audio assessment
• Model loading and inference functionality
• Generation capability verification

Results

The model successfully generates Nigerian-accented English speech and maintains the multi-speaker capabilities of the base model. However, extended training would be beneficial for production use.

Summary

This model serves as a proof-of-concept for Nigerian English TTS and demonstrates the effectiveness of LoRA fine-tuning for speech synthesis models.

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA Tesla T4 GPU
• Hours used: ~0.1 hours (2-5 minutes)
• Cloud Provider: Google Colab
• Compute Region: Variable
• Carbon Emitted: Minimal due to short training time

Technical Specifications

Model Architecture and Objective

• Architecture: CSM (Conditional Speech Model) - transformer-based text-to-speech model
• Objective: Conditional generation of speech audio from text with speaker conditioning
• Input: Text tokens + speaker ID
• Output: Audio waveform at 24kHz sampling rate
• Fine-tuning method: LoRA (Low-Rank Adaptation)

Compute Infrastructure

Training was conducted using the Unsloth framework for memory-efficient fine-tuning.

Hardware

• Google Colab Tesla T4 GPU (16GB VRAM)
• CUDA-compatible environment
• Memory-optimized training with Unsloth

Software

• Python 3.10+
• PyTorch 2.0+
• Transformers 4.52.3
• Unsloth framework
• Additional dependencies: soundfile, datasets, accelerate

Citation

BibTeX:

Glossary

• CSM: Conditional Speech Model - A transformer-based architecture for text-to-speech synthesis
• LoRA: Low-Rank Adaptation - An efficient fine-tuning method that updates only a small subset of model parameters
• TTS: Text-to-Speech - Technology that converts written text into spoken audio
• Nigerian English: The variety of English spoken in Nigeria, with distinct pronunciation and linguistic features

Model Card Authors

Sam4rano

Model Card Contact

For questions or issues regarding this model, please create an issue in the model repository or contact through Hugging Face.