Bharat-Vidya

StyleTTS2 Bilingual TTS with Voice Cloning

End-to-end Text-to-Speech system with voice cloning for English and Gujarati using StyleTTS2 architecture.

Features

• ✅ Bilingual Support: English and Gujarati
• ✅ Voice Cloning: Generate speech in any voice from a reference audio
• ✅ StyleTTS2 Architecture: State-of-the-art diffusion-based TTS
• ✅ Domain Embeddings: Pseudo-speaker variations from different domains
• ✅ REST API: FastAPI server for production deployment
• ✅ Pretrained Vocoder: HiFi-GAN for high-quality audio generation

Architecture

Components:

• Text Encoder: Phoneme-based Transformer encoder
• Style Encoder: Reference encoder for speaker embedding extraction
• Duration Predictor: MLP for phoneme duration prediction
• Diffusion Decoder: Denoising diffusion probabilistic model (DDPM)
• Vocoder: Pretrained HiFi-GAN V1

Installation

# Clone repository
git clone https://github.com/yourusername/styletts2-bilingual.git
cd styletts2-bilingual/styletts2_tts

# Install dependencies
pip install -r requirements.txt

# Install package
pip install -e .

Dependencies

• Python 3.9+
• PyTorch 2.0+
• torchaudio 2.0+
• librosa, soundfile
• phonemizer (requires espeak-ng)
• FastAPI, uvicorn
• tensorboard

Download HiFi-GAN Vocoder

Download the pretrained HiFi-GAN V1 model:

# Option 1: From official repository
wget https://github.com/jik876/hifi-gan/releases/...

# Option 2: From Google Drive
# Download from: https://drive.google.com/uc?id=1-eEYTB5Av9jNql0WGBlRoi-WH2J7bp5Y

# Place in: pretrained/hifigan/

Dataset

SYSPIN Dataset (IISc SPICOR):

• English: 25,158 samples (47h 52min), 1 male speaker
• Gujarati: 7,834 samples (23h 38min), 1 male speaker

Dataset Structure:

data/
├── english/
│   ├── wav/                    # 25,158 WAV files (44.1kHz)
│   └── IISc_SPICORProject_English_Male_Spk001_HC_Transcripts.json
└── gujarati/
    ├── wav/                    # 7,834 WAV files (44.1kHz)
    └── IISc_SPICORProject_Gujarati_Male_Spk001_HC_Transcripts.json

Usage

1. Data Preprocessing

# English
python src/preprocessing/prepare_data.py \
  --config configs/english_config.yaml

# Gujarati
python src/preprocessing/prepare_data.py \
  --config configs/gujarati_config.yaml

What this does:

• Loads raw SYSPIN data
• Resamples audio: 44.1kHz → 22.05kHz
• Extracts mel-spectrograms
• Phonemizes text (or character-level encoding)
• Splits into train/val/test
• Saves processed metadata

2. Training

# English
python src/training/train.py \
  --config configs/english_config.yaml

# Gujarati
python src/training/train.py \
  --config configs/gujarati_config.yaml

Training Time:

• English: ~80-100 hours on A100 (40GB)
• Gujarati: ~60-80 hours on A100

Monitoring:

tensorboard --logdir logs/

3. Inference

Standalone Script:

python inference/synthesize.py \
  --text "Hello world" \
  --ref_audio reference.wav \
  --output output.wav \
  --config configs/english_config.yaml \
  --checkpoint checkpoints/english/best.pt

API Server:

# Start server
python api_server.py

# Test API
curl -X GET "http://localhost:7860/Get_Inference" \
  -F "text=Hello world" \
  -F "lang=english" \
  -F "[email protected]" \
  -o output.wav

API Endpoints

GET /Get_Inference

Generate speech from text with voice cloning.

Parameters:

• text (str): Text to synthesize
• lang (str): Language (english or gujarati)
• speaker_wav (file): Reference audio for voice cloning

Returns: Audio file (WAV format)

Example:

import requests

with open('reference.wav', 'rb') as f:
    response = requests.get(
        'http://localhost:7860/Get_Inference',
        params={'text': 'Hello world', 'lang': 'english'},
        files={'speaker_wav': f}
    )

with open('output.wav', 'wb') as f:
    f.write(response.content)

GET /health

Health check endpoint.

Returns:

{
  "status": "healthy",
  "models_loaded": 2,
  "languages": ["english", "gujarati"],
  "device": "cuda"
}

Configuration

Key Settings (configs/english_config.yaml):

# Audio
audio:
  sample_rate: 22050
  n_mels: 80

# Model
model:
  text_encoder:
    vocab_size: 100
    hidden_dim: 256
    num_layers: 6

  diffusion:
    diffusion_steps: 1000
    inference_steps: 50
    noise_schedule: cosine

# Training
training:
  batch_size: 16
  learning_rate: 0.0001
  max_steps: 500000
  mixed_precision: true

Project Structure

styletts2_tts/
├── configs/
│   ├── english_config.yaml
│   └── gujarati_config.yaml
├── src/
│   ├── models/               # Model architecture
│   ├── preprocessing/        # Data processing
│   ├── training/            # Training loop
│   └── utils/               # Utilities
├── data/                    # Dataset (not included)
├── checkpoints/             # Model checkpoints
├── pretrained/              # Pretrained vocoder
├── logs/                    # TensorBoard logs
├── api_server.py           # FastAPI server
├── requirements.txt
└── README.md

Design Decisions

Troubleshooting

espeak-ng not installed

# Windows
Download from: https://github.com/espeak-ng/espeak-ng/releases

# Linux
sudo apt-get install espeak-ng

# macOS
brew install espeak-ng

Out of memory

• Reduce batch_size in config
• Enable mixed_precision: true
• Use gradient accumulation
• Reduce model hidden_dim

Vocoder not found

Download HiFi-GAN and place in pretrained/hifigan/:

pretrained/hifigan/
├── generator_v1           # PyTorch checkpoint
└── config.json           # Vocoder config

Citation

@misc{styletts2-bilingual,
  title={StyleTTS2 Bilingual TTS with Voice Cloning},
  author={SYSPIN TTS Competition},
  year={2025},
  howpublished={\url{https://github.com/yourusername/styletts2-bilingual}}
}

License

MIT License

Acknowledgments

• StyleTTS2: https://github.com/yl4579/StyleTTS2
• HiFi-GAN: https://github.com/jik876/hifi-gan
• SYSPIN Dataset: IISc SPICOR Project

---

Status: ✅ Ready for preprocessing and training!

For questions or issues, please open a GitHub issue.