chatterbox-bangla

ChatterBox Bangla

A fine-tuned version of ResembleAI/chatterbox for Bengali (Bangla) text-to-speech synthesis.

Zerohot TTS Output

Model Details

• Base model: ResembleAI/chatterbox — multilingual ChatterBox (supports 23 languages)
• Fine-tuned on: Bengali speech corpus (~99 hours, 58,820 samples)
  • ai4bharat/Shrutilipi (Bengali split) — 17,882 samples, ~28h
  • ai4bharat/Rasa (Bengali split) — 28,088 samples, ~51h
  • SPRINGLab/IndicTTS_Bengali — 12,852 samples, ~20h
• Training steps: 20,000
• Epochs: ~10.6
• Architecture: T3 (Text-to-Token Transformer) + HiFT-GAN vocoder
• Vocabulary: Extended from 704 → 2,530 tokens to cover Bengali characters

Requirements

git clone https://github.com/gokhaneraslan/chatterbox-finetuning
cd chatterbox-finetuning
pip install -r requirements.txt

One-time patch (upstream vocab resize fix)

The upstream chatterbox-finetuning repo initialises T3 with a hard-coded 704-token vocabulary, which causes a size mismatch when loading this model (vocab=2530). Apply this one-line fix before running inference:

# Run from inside the cloned chatterbox-finetuning directory
python - <<'EOF'
import pathlib
f = pathlib.Path("src/chatterbox_/tts.py")
txt = f.read_text()
old = "        t3 = T3()\n        t3_state = load_file(ckpt_dir / \"t3_cfg.safetensors\")"
new = (
    "        t3_state = load_file(ckpt_dir / \"t3_cfg.safetensors\")\n"
    "        from .models.t3.modules.t3_config import T3Config\n"
    "        t3 = T3(hp=T3Config(text_tokens_dict_size=t3_state[\"text_emb.weight\"].shape[0]))"
)
f.write_text(txt.replace(old, new))
print("Patched tts.py")
EOF

Usage

import sys
sys.path.insert(0, "/path/to/chatterbox-finetuning")

from huggingface_hub import snapshot_download
from src.chatterbox_.tts import ChatterboxTTS
import torchaudio

model_dir = snapshot_download("BosonLab/chatterbox-bangla")
model = ChatterboxTTS.from_local(model_dir, device="cuda")

text = "আমি বাংলায় কথা বলতে পারি। এটি একটি পরীক্ষামূলক বাক্য।"
wav = model.generate(text)

torchaudio.save("output.wav", wav, model.sr)

With Voice Cloning

wav = model.generate(text, audio_prompt_path="reference.wav")

Files

Training Data

All audio resampled to 16kHz. Text cleaned and normalized. Datasets sourced from AI4Bharat and SPRINGLab public datasets.

Limitations

• Optimized for Bengali; other languages may degrade
• Best results with clear, well-punctuated Bengali text
• Emotion control inherited from base ChatterBox multilingual model
• Requires chatterbox-finetuning kit due to extended Bengali vocabulary