TTS_FINETUNE_ENGLISH

Overview : This project implements a Text-to-Speech (TTS) system using the SpeechT5 model fine-tuned on the LJ Speech dataset. The goal is to generate natural-sounding speech from textual input, leveraging state-of-the-art machine learning techniques and for english technical words.

Link to the code: https://github.com/lithish262004/TTS_FINETUNE_ENGLISH

Methodology This section outlines the detailed steps taken for model selection, dataset preparation, and fine-tuning of the Text-to-Speech (TTS) system using the SpeechT5 model.

1. Model Selection Model Choice: SpeechT5

SpeechT5 is a transformer-based model designed for TTS tasks. It provides a balance of performance and efficiency, making it suitable for generating high-quality speech. We selected SpeechT5 due to its ability to handle various speech generation tasks, including text-to-speech synthesis, making it versatile for our project. 2. Dataset Preparation Dataset: LJ Speech

The LJ Speech dataset comprises approximately 13,100 audio clips of a single speaker reading passages from various texts. It includes: Audio Files: WAV format audio samples. Transcripts: Text files containing the spoken text for each audio sample. Preparation Steps:

Data Download: Download the LJ Speech dataset from keithito.com

Preprocessing:

Audio Processing: Convert all audio files to a consistent format (16000 Hz sample rate, mono channel). Normalize audio levels to ensure consistent volume across samples. Text Cleaning: Remove any extraneous characters or formatting from the transcripts to ensure clean input for the model. Optionally, apply phonetic transcriptions for improved pronunciation. Alignment:

Generate a mapping between audio files and their corresponding text transcripts, ensuring that each audio clip can be paired with its correct spoken text. Train-Validation Split:

Split the dataset into training and validation sets (e.g., 90% training, 10% validation) to evaluate model performance during and after fine-tuning. 3. Fine-Tuning Fine-Tuning Process:

Environment Setup:

Ensure that all necessary libraries and dependencies (e.g., PyTorch, Transformers, NumPy) are installed as specified in the requirements.txt file. Fine-Tuning Script:

Model loading: Load the pre-trained SpeechT5 model. Data loading: Use a data loader to read audio and text pairs for training. Training loop: Implement a loop that iterates over the training dataset for a specified number of epochs, updating model weights based on the loss calculated from predictions. Hyperparameter Configuration:

Set hyperparameters such as learning rate, batch size, and the number of epochs. Commonly used values might include: Learning Rate: 1e-5 Batch Size: 16 Epochs: 10/11 Monitoring:

Monitor training loss and validation metrics during the fine-tuning process to prevent overfitting. Use techniques like early stopping if validation loss starts to increase. Model Saving:

After fine-tuning, save the trained model and any associated artifacts (e.g., tokenizer) for later use in generating speech. 4. Evaluation Post-Fine-Tuning Evaluation:

After fine-tuning, evaluate the model using the validation dataset. Metrics for evaluation may include: Mean Opinion Score (MOS): A subjective score based on human evaluations of audio quality. Alignment and accuracy: Check how well the generated speech aligns with the input text

TRAINING PROGRESS:

Step Training Loss Validation Loss
1000 0.437300 0.389510

                                     2000	0.411300	    0.379249
                                     3000	0.415100	    0.376344
                                    4000	0.416200	    0.375081

Environment and Dependencies Transformers: 4.44.2 PyTorch: 2.4.1+cu121 Datasets: 3.0.1 Tokenizers: 0.19.1

RESULTS:

Results In this section, we present the results of our Text-to-Speech (TTS) system, including both objective and subjective evaluations. We tested the model's performance on two types of speech: English technical speecH

Objective Evaluations Metrics Used: Mean Opinion Score (MOS): A numerical measure of perceived audio quality, usually rated on a scale from 1 (poor) to 5 (excellent). English Technical Speech Subjective evaluations were conducted through listener studies, where participants rated the audio samples generated by the model. The evaluations focused on clarity, naturalness, and overall satisfaction. English Technical Speech Feedback Clarity: Most listeners appreciated the clarity of the speech, especially in technical contexts. Naturalness: While the speech was generally natural, some participants felt that the intonation could be improved to sound more conversational. Comments: “Very clear for technical topics, but sometimes feels robotic.

Challenges and Solutions: Dataset Challenges Limited availability of high-quality ENGLISH speech data with technical terms

Conclusion:

Successfully fine-tuned SpeechT5 for english language

Maintained high quality while optimizing performance

Future Improvements: Expand dataset with more diverse speakers

Further optimize inference speed

Recommendations:

Regular model retraining with expanded datasets

Integration of automated quality assessment tools