Nos_TTS-paulo-vits-phonemes

Paulo: Nos Project's Galician TTS Model

Model description

Paulo is a Galician TTS model developed by the Nós project. It was trained from scratch using the Coqui TTS Python library on the Paulo corpus of the dataset CRPIH_UVigo-GL-Voices. This corpus comprises a total of 1,316 sentences recorded by an amateur voice talent.

The model was trained on phonemes input, so it needs a phonetic transcription of the input text to synthesize speech. The Cotovía tool is required to generate this transcription. The Cotovía linguistic module also performs text normalization as a preliminary step to phonetic transcription.

You can test the model in our live inference demo (Nós-TTS).

Intended uses and limitations

You can use this model to generate synthetic speech in Galician.

Installation

Cotovía

For phonetic transcription, the Cotovía front-end must be used. This software is available for download on the SourceForge website. The required Debian packages are cotovia_0.5_amd64.deb and cotovia-lang-gl_0.5_all.deb, which can be installed using the following commands:

sudo dpkg -i cotovia_0.5_amd64.deb
sudo dpkg -i cotovia-lang-gl_0.5_all.deb

TTS library

To synthesize speech, you need to install the Coqui TTS library:

pip install TTS

How to use

<!-- ### Command-line usage The tool can be used to generate the phonetic transcription of the text. The following command can be used to generate the phonetic transcription of a text string: ```bash echo "Era unha avioneta... O piloto era pequeno, que se chega a ser dos grande, tómbate!" | cotovia -t -n -S | iconv -f iso88591 -t utf8 ``` -->

Python usage

Normalization and synthesis can also be performed within Python:

import argparse
import random
import re
import string
import subprocess
from TTS.utils.synthesizer import Synthesizer

PUNCLIST = [';', '?', '¿', ',', ':', '.', '!', '¡']

def sanitize_filename(filename):
    """Remove or replace any characters that are not allowed in file names."""
    return ''.join(c for c in filename if c.isalnum() or c in (' ', '_', '-')).rstrip()

def canBeNumber(n):
    try:
        int(n)
        return True
    except ValueError:
        # Not a number
        return False

def is_number(index, text):
    if index == 0:
        return False
    elif index == len(text) - 1:
        return False
    else:
        return canBeNumber(text[index - 1]) and canBeNumber(text[index + 1])

def split_punc(text):
    segments = []
    puncs = []
    curr_seg = ""
    previous_punc = False

    for i, c in enumerate(text):        
        if c in PUNCLIST and not previous_punc and not is_number(i, text):
            segments.append(curr_seg.strip())
            puncs.append(c)
            curr_seg = ""
            previous_punc = True
        elif c in PUNCLIST and previous_punc:
            puncs[-1] += c
        else:
            curr_seg += c
            previous_punc = False

    segments.append(curr_seg.strip())

    #Remove empty segments in the list
    segments = filter(None, segments)

    # store segments as a list
    segments = list(segments)

    return segments, puncs

def remove_tra3_tags(phontrans):
    s = re.sub(r'#(.+?)#', r'', phontrans)
    s = re.sub(r'%(.+?)%', r'', s)
    s = re.sub(' +',' ',s)
    s = re.sub('-','',s)
    return s.strip()

def to_cotovia(text_segments):
    # Input and output Cotovía files
    res = ''.join(random.choices(string.ascii_lowercase + string.digits, k=5))
    COTOVIA_IN_TXT_PATH = res + '.txt'
    COTOVIA_IN_TXT_PATH_ISO = 'iso8859-1' + res + '.txt'
    COTOVIA_OUT_PRE_PATH = 'iso8859-1' + res + '.tra'
    COTOVIA_OUT_PRE_PATH_UTF8 = 'utf8' + res + '.tra'
    
    with open(COTOVIA_IN_TXT_PATH, 'w') as f:
        for seg in text_segments:
            if seg:
                f.write(seg + '\n')
            else:
                f.write(',' + '\n')

    # utf-8 to iso8859-1
    subprocess.run(["iconv", "-f", "utf-8", "-t", "iso8859-1", COTOVIA_IN_TXT_PATH, "-o", COTOVIA_IN_TXT_PATH_ISO], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
    # call cotovia with -t3 option
    subprocess.run(["cotovia", "-i", COTOVIA_IN_TXT_PATH_ISO, "-t3", "-n"], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
    # iso8859-1 to utf-8
    subprocess.run(["iconv", "-f", "iso8859-1", "-t", "utf-8", COTOVIA_OUT_PRE_PATH, "-o", COTOVIA_OUT_PRE_PATH_UTF8], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

    segs = []
    try:
        with open(COTOVIA_OUT_PRE_PATH_UTF8, 'r') as f:
            segs = [line.rstrip() for line in f]
            segs = [remove_tra3_tags(line) for line in segs]
    except:
        print("ERROR: Couldn't read cotovia output")

    subprocess.run(["rm", COTOVIA_IN_TXT_PATH, COTOVIA_IN_TXT_PATH_ISO, COTOVIA_OUT_PRE_PATH, COTOVIA_OUT_PRE_PATH_UTF8], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

    return segs

def merge_punc(text_segs, puncs):
    merged_str = ""

    for i, seg in enumerate(text_segs):
        merged_str += seg + " "
        if i < len(puncs):
            merged_str += puncs[i] + " "

    # remove spaces before , . ! ? ; : ) ] of the merged string
    merged_str = re.sub(r"\s+([.,!?;:)\]])", r"\1", merged_str)

    # remove spaces after ( [ ¡ ¿ of the merged string
    merged_str = re.sub(r"([\(\[¡¿])\s+", r"\1", merged_str)

    return merged_str.strip()


def accent_convert(phontrans):
    transcript = re.sub('a\^','á',phontrans)
    transcript = re.sub('e\^','é',transcript)
    transcript = re.sub('i\^','í',transcript)
    transcript = re.sub('o\^','ó',transcript)
    transcript = re.sub('u\^','ú',transcript)
    transcript = re.sub('E\^','É',transcript)
    transcript = re.sub('O\^','Ó',transcript)
    return transcript


def text_preprocess(text):
    #Split from punc
    text_segments, puncs = split_punc(text)
    cotovia_phon_segs = to_cotovia(text_segments)
    cotovia_phon_str = merge_punc(cotovia_phon_segs, puncs)
    phon_str = accent_convert(cotovia_phon_str)
    return phon_str


def main():
    parser = argparse.ArgumentParser(description='Cotovia phoneme transcription.')
    parser.add_argument('text', type=str, help='Text to synthetize')
    parser.add_argument('model_path', type=str, help='Absolute path to the model checkpoint.pth')
    parser.add_argument('config_path', type=str, help='Absolute path to the model config.json')

    args = parser.parse_args()

    print("Text before preprocessing: ", args.text)
    text = text_preprocess(args.text)
    print("Text after preprocessing: ", text)
    synthesizer = Synthesizer(
        args.model_path, args.config_path, None, None, None, None,
    )

    # Step 1: Extract the first word from the text
    first_word = args.text.split()[0] if args.text.split() else "audio"
    first_word = sanitize_filename(first_word)  # Sanitize to make it a valid filename

    # Step 2: Use synthesizer's built-in function to synthesize and save the audio
    wavs = synthesizer.tts(text)
    filename = f"{first_word}.wav"
    synthesizer.save_wav(wavs, filename)

    print(f"Audio file saved as: {filename}")

if __name__ == "__main__":
    main()

This Python code takes a text input, preprocesses it using Cotovía's front-end, synthesizes speech from the preprocessed text, and saves the synthetic output speech as a .wav file.

A more advanced version, including additional text preprocessing, can be found in the script synthesize.py, avaliable in this repository. You can use this script to synthesise speech from an input text as follows:

python synthesize.py text model_path config_path

Training

Hyperparameter

The model is based on VITS proposed by Kim et al. The following hyperparameters were set in the coqui framework.

The model was trained for 610,000 steps.

The nos_fonemas formatter is a modification of the LJSpeech formatter with one extra column for the phonemized input.

Additional information

Authors

Antonio Moscoso and Carmen Magariños.

Contact information

For further information, send an email to [email protected]

Funding

This research was produced within the framework of the Proxecto Nós, funded by the Ministry for Digital Transformation and Public Administration and the Recovery, Transformation, and Resilience Plan – Funded by the European Union – NextGenerationEU, as part of the Ilenia Project with reference 2022/TL22/00215336, and previously “The Nós project: Galician in the society and economy of Artificial Intelligence”, resulting from the agreement 2021-CP080 between the Xunta de Galicia and the University of Santiago de Compostela, and thanks to the Investigo program, within the National Recovery, Transformation and Resilience Plan, within the framework of the European Recovery Fund (NextGenerationEU).

Citation information

If you use this model, please cite as follows:

Moscoso, Antonio; Magariños, Carmen. 2024. Nos_TTS-paulo-vits-phonemes. URL: https://huggingface.co/proxectonos/Nos_TTS-paulo-vits-phonemes/