TTS_T5-uz_ru_en

TTS_T5-uz_ru_en Model Card

Model Description

TTS_T5-uz_ru_en is a multilingual text-to-speech (TTS) synthesis system built upon a modified encoder-decoder T5 architecture template. Developed using the orchestration engine in the companion firdavsus/T5-speech GitHub repository.

The framework provides robust cross-lingual acoustic stability across three target languages: Uzbek (uz), Russian (ru), and English (en).

voice-clonning (UZ, RU, EN)

Base voice -> Listen to the Audio Track

Model Features & Specifications

• Task: Text-to-Speech / Text-to-Audio Synthesis
• Core Engine: Sequence-to-sequence T5 (Text-to-Text Transfer Transformer) adaptation configured to map text tokens to target speech representations.
• Languages Supported: Uzbek, Russian, and English.
• Output Profiles: Compatible with neural vocoders (e.g., HiFi-GAN) or autoregressive audio token decoders to synthesize raw 16kHz/24kHz waveforms.

---

Architectural Workflow

The model re-imagines speech synthesis as a standard sequence-to-sequence translation task:

1. Phoneme/Text Encoding: The T5 text encoder processes character sequences, expanding them with cross-lingual language tags to preserve regional accents.
2. Acoustic Decoding: The autoregressive decoder generates sequential acoustic frames (such as mel-spectrogram slices) or discrete codebook indices conditioned on the encoder's hidden representations.
3. Vocoding Stage: Generated representations are passed to a neural vocoder block to output natural-sounding human audio speech.

---

Intended Uses & Limitations

Target Applications

• Multilingual Assistant Platforms: Powering interactive voice agents or accessibility tools requiring fluent localized speech in Uzbekistan.
• Speech Synthesis Research: Investigating text-to-speech capabilities within encoder-decoder LLM frameworks without separating the acoustic and text pipelines.
• Audio Tokenization: Creating down-stream audio tokens suitable for multimodal or speech-to-speech language modeling.

Limitations

• Prosody Control: Fine-grained pacing variations (e.g., intense whispering or shouting) may require explicit token prefixing depending on the configuration layer chosen during training.
• Vocoder Dependency: The raw audio fidelity depends directly on the vocoder setup applied to decode the model's structural audio predictions.

---

Quickstart Inference

You can run sequence generation and audio rendering loops using the evaluation pipeline scripts available in the primary GitHub repository.

import torch
from model import T5SpeechGenerator  # Imported from your firdavsus/T5-speech repository
import scipy.io.wavfile as wavfile

# 1. Initialize the multilingual TTS pipeline
device = "cuda" if torch.cuda.is_available() else "cpu"
model = T5SpeechGenerator.from_pretrained("firdavsus/TTS_T5-uz_ru_en").to(device)

# 2. Prepare text input with language formatting prefix
text_prompt = "<uz> Assalomu alaykum! Bugun ob-havo juda yaxshi."

# 3. Generate raw audio waveforms
with torch.no_grad():
    audio_outputs = model.generate_speech(text_prompt)

# 4. Save synthesized audio track to local disk
wavfile.write("output_uzbek.wav", rate=24000, data=audio_outputs.cpu().numpy())
print("Multilingual speech successfully synthesized and exported to disk.")