Nos_StyleTTS2-Brais-GL

Nos_StyleTTS2-Brais-GL: Galician Male TTS Model

Overview

• Model Description
• System Architecture and Components
• Intended Uses and Limitations
• Prerequisites and Environment Setup
• Auxiliary Components
• Model Training
• Inference
• Evaluation
• Results
• Citation
• References
• Additional Information
• Funding and Acknowledgements

---

Model Description

Nos_StyleTTS2-Brais-GL is a high-quality text-to-speech (TTS) model for Galician based on the StyleTTS2 architecture. It was trained on the single-speaker male voice "Brais" (proxectonos/Nos_Brais-GL), developed as part of Proxecto Nós.

This model integrates a native Galician processing pipeline, utilizing Cotovía for grapheme-to-phoneme (G2P) conversion and PL-ModernBERT-gl as its phoneme-level language encoder, providing rich contextualized phoneme embeddings.

Key Features:

• Architecture: StyleTTS2 with iSTFTNet decoder, diffusion-based style modeling, and Speech Language Model (SLM) adversarial training via HuBERT.
• Phoneme Encoder: Powered by proxectonos/PL-ModernBERT-gl (ModernBERT trained on Galician phonemes).
• G2P & Normalization: Native Galician text processing using Cotovía.
• Voice Profile: Male Galician voice ("Brais") based on proxectonos/Nos_Brais-GL.

---

System Architecture and Components

The synthesis pipeline relies on four main modules:

1. Phoneme Encoder (PL-BERT): Uses proxectonos/PL-ModernBERT-gl trained with Cotovía's phoneme dictionary to supply contextual embeddings.
2. Auxiliary ASR Aligner: Text-audio alignment trained using a modified version of AuxiliaryASR adapted for Cotovía phonemes.
3. Pitch Extractor (JDC): Pre-trained Joint Detection and Classification (JDC) network for F0 pitch extraction.
4. SLM Discriminator: Uses BSC-LT/hubert-base-los-2k to calculate Speech Language Model losses during second-stage adversarial training.

---

Intended Uses and Limitations

Intended Uses

• High-quality Galician speech synthesis for a wide range of applications (e.g., accessibility tools, virtual assistants, conversational agents, entertainment).
• Research on prosody modeling and speech synthesis in low-resource and regional language contexts.

Limitations

• Single-speaker model: voice characteristics are fixed to the "Brais" male voice profile.
• Requires the Cotovía phonemizer for G2P conversion.

---

Prerequisites and Environment Setup

1. Clone this repository:

git clone [https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL](https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL)
cd Nos_StyleTTS2-Brais-GL

1. Set up the virtual environment and dependencies:

conda create -n StyleTTS2 python=3.10 -y
conda activate StyleTTS2
pip install -r requirements.txt

1. Install Cotovía: Cotovía is required for grapheme-to-phoneme (G2P) conversion. Follow the installation steps provided in Utils/cotovia/README.md.

Dataset Preparation

Training data can be downloaded using the provided download script. You can specify the primary training dataset and the Out-Of-Domain (OOD) dataset in Configs/download_data.yml.

To download the dataset via the script:

python scripts/download_data.py --config Configs/download_data.yml --download_data

Note: If you wish to switch the primary dataset, keep the existing audio folders, delete the .txt annotation files, and rerun the script without the --download_data flag:

python scripts/download_data.py --config Configs/download_data.yml

Auxiliary Components

To ensure optimal synthesis, StyleTTS2 relies on three auxiliary models aligned with the same phonemizer (Cotovía):

1. Phoneme Encoder (PL-BERT)

StyleTTS2 requires a PL-BERT model trained on Galician phonemes generated by Cotovía. The training code and utilities for this component are maintained in the proxectonos/PL-ModernBERT-gl repository.

2. Text-Audio Aligner (Auxiliary ASR)

An Auxiliary ASR model aligns phoneme sequences with acoustic features. Follow the instructions in Utils/ASR/AuxiliaryASR (a modified version of AuxiliaryASR adapted for Cotovía). You can reuse the dataset downloaded in the previous steps for retraining.

3. Pitch Extractor (JDC)

The Utils/JDC directory contains a pre-trained Joint Detection and Classification (JDC) model for pitch (F0) extraction. Although pre-trained on the English LibriTTS corpus, pitch dynamics are language-independent, making retraining unnecessary.

Model Training

Configuration Parameters

Paths & Checkpoints

Important: For PLBERT_dir, specify only the directory containing the config.yml configuration file, which indicates the path to the BERT model.

Main Training Parameters

Decoder, SLM, and Diffusion Pipeline

<!-- Extended Configuration Parameters (Full view from config.yml) #### Main Training Parameters (Extended) | Parameter | Value | | :--- | :--- | | `epochs_1st` | 100 | | `epochs_2nd` | 100 | | `batch_size` | 4 | | `grad_accumulation` | 1 | | `max_len` | 500 | | `load_only_params` | false | | `preprocess.sr` | 24000 | | `preprocess.n_fft` | 2048 | | `preprocess.win_length` | 1200 | | `preprocess.hop_length` | 300 | | `model.multispeaker` | false | | `model.dim_in` | 64 | | `model.hidden_dim` | 512 | | `model.max_conv_dim` | 512 | | `model.n_layer` | 3 | | `model.n_mels` | 80 | | `model.n_token` | 69 | | `model.max_dur` | 50 | | `model.style_dim` | 128 | | `model.dropout` | 0.2 | #### Decoder, SLM, and Diffusion (Extended) | Parameter | Value | | :--- | :--- | | `decoder.type` | istftnet | | `decoder.resblock_kernel_sizes` | [3,7,11] | | `decoder.upsample_rates` | [10,6] | | `decoder.upsample_initial_channel` | 512 | | `decoder.resblock_dilation_sizes` | [[1,3,5],[1,3,5],[1,3,5]] | | `decoder.upsample_kernel_sizes` | [20,12] | | `decoder.gen_istft_n_fft` | 20 | | `decoder.gen_istft_hop_size` | 5 | | `slm.model` | BSC-LT/hubert-base-los-2k | | `slm.sr` | 16000 | | `slm.hidden` | 768 | | `slm.nlayers` | 13 | | `slm.initial_channel` | 64 | | `diffusion.embedding_mask_proba` | 0.1 | | `diffusion.transformer.num_layers` | 3 | | `diffusion.transformer.num_heads` | 8 | | `diffusion.transformer.head_features` | 64 | | `diffusion.transformer.multiplier` | 2 | -->

---

Executing Training

Training is executed in two consecutive stages:

1. First Stage Training (Acoustic Base):

accelerate launch train_first.py --config_path ./Configs/config.yml

1. Second Stage Training (Diffusion & SLM Refinement):

python train_second.py --config_path ./Configs/config.yml

Note: Multi-GPU training is not supported in the second stage.

Checkpoints are saved in log_dir using the naming conventions epoch_1st_%05d.pth and epoch_2nd_%05d.pth.

---

Inference

This repository includes an inference script to synthesize audio using a pre-trained StyleTTS2-GL model. It accepts raw Galician text strings or input text files and supports configurable parameters to control the generation process.

python inference.py --config Configs/inference_config.yml --text "Texto en galego para sintetizar." --device 0

Command-Line Arguments

Recommended Inference Settings

The following parameters are recommended for optimal audio synthesis using the Brais StyleTTS2 model:

---

Evaluation

An evaluation script utilizing the speechmos library is included to assess audio quality and verify model performance via MOS (Mean Opinion Score).

To run the evaluation on generated audio files:

python scripts/eval.py --audios audio1.wav audio2.wav --output output/mos_results.txt

Depending on the number of audio files provided, the script performs the following calculations:

• MOS Calculation: Performed for each individual audio file listed.
• CMOS (Comparative MOS): Automatically calculated when exactly two audio files are passed to compare performance against a reference model.

Results will be printed to the stdout console and saved in the designated --output file.

Results

Results were obtained using the speechmos library to measure predicted DNSMOS (OVRL) quality and calculate CMOS gains.

Text Samples

Models were evaluated using three different text lengths containing a mix of neutral, interrogative, and exclamatory sentences:

• Short: ~10 seconds of generated audio.
• Medium: ~30 seconds of generated audio.
• Long: >60 seconds of generated audio.

Synthesized audio samples were benchmarked against original corpus recordings and baseline VITS models from Proxecto Nós.

Original Corpus Reference Score

Synthesis Quality Comparison (Brais Voice)

Citation

If this model contributes to your research, please cite it as follows:

@misc{proxectonos/Nos_StyleTTS2-Brais-GL,
  author       = {{Proxecto Nós}},
  title        = {{Nos_StyleTTS2-Brais-GL: Galician Male TTS Model}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{[https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL](https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL)}},
}

References

Models

• AuxiliaryASR
• PL-Bert
• StyleTTS2

Datasets

• Galician AI Language Resources (Zenodo)
• TTS datasets - Proxecto Nós (Hugging Face)
• ASR datasets - Proxecto Nós (Hugging Face)

Additional Information

Licensing

This model is licensed under the Apache License 2.0.

Authors and Credits

• Project Oversight: Proxecto Nós
• Technical Development: Gradiant (Centro Tecnolóxico de Telecomunicacións de Galicia)

Funding and Acknowledgements

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.

We would like to express our gratitude to the engineering and research teams at Gradiant for the technical development of this model, as well as to the Aholab Signal Processing Laboratory (HiTZ) and the Language Technologies Laboratory (BSC) for their technical support and collaboration.