ttslibre-poc-es-en

TTSLibre – Spanish + English proof of concept (experiment 016)

Proof-of-concept checkpoint from TTSLibre, an open recipe for a small zero-shot TTS (Supertonic-lineage: character input, flow matching over a compressed autoencoder latent, Vocos vocoder). ~19.8M parameters, trained from scratch on an interleaved 50/50 mix of 4 h Argentinian Spanish (OpenSLR 61) and 4 h English (LibriTTS-R), fewest speakers per language. Same architecture and code as ttslibre-poc-en and ttslibre-poc-es-ar, plus language tags: every text is wrapped in <es>...</es> or <en>...</en> (4 single-token tags appended to the vocab, 80 -> 84). Not a usable product: intermediate checkpoint of a run still in progress, chosen by wall time, no whole-val evaluation (see the repo's experiment 016).

Files

Vocoder: charactr/vocos-mel-24khz (loaded from the Hub at runtime).

Training

4.0 h OpenSLR 61 es-AR + 4.0 h LibriTTS-R English, rows alternating es/en, 5% held out, 1x RTX 3090, batch 32-48, lr 5e-4, random init. Probe WER during training (Whisper, 1 draw per sentence per round, Spanish reference voice for both languages; heldout = 3 val sentences per language, novel = hand-written):

Reference: the Spanish-only PoC (014, 8 h) had held-out 0.38 at 2h21 and 0.10 at 3h30.

Inference from a cold start

Tested with Python 3.12, torch 2.10 (CUDA 12.8), GPU or CPU. The language is detected from the text (Spanish characters / stopwords; default English) and tagged automatically; pass lang="es"/"en" to force it.

git clone https://github.com/franciscocarloserra/ttslibre && cd ttslibre
python3 -m venv venv && . venv/bin/activate
pip install torch torchaudio vocos soundfile numpy num2words einops huggingface_hub
hf download FranciscoCarlosErra/ttslibre-poc-es-en --local-dir hfckpt
E=experiments/20-scaling/016-mixed-es-en
mkdir -p $E/runs/esen $E/runs/ae datasets/mixed-es-en/prep datasets/openslr61
cp hfckpt/ttl.pt $E/runs/esen/ttl.pt
cp hfckpt/ae.pt $E/runs/ae/ae.pt
cp hfckpt/latent_stats.pt hfckpt/vocab.json datasets/mixed-es-en/prep/
cp hfckpt/ref_default.wav datasets/openslr61/arf_00295_00000740990.wav
cp hfckpt/voices/*.pt experiments/voices/
cd $E
python synth.py "Mañana vamos a la playa con los chicos." hola.wav --run runs/esen/ttl.pt --ref ../../voices/arm_06136.pt
python synth.py "The quick brown fox jumps over the lazy dog." fox.wav --run runs/esen/ttl.pt --ref ../../voices/arf_06136.pt
# add --cpu for CPU; --ref can also be any 5-10 s wav of a new speaker

In-memory use (load once, generate many)

import sys, soundfile as sf
sys.path.insert(0, "experiments/20-scaling/016-mixed-es-en")
from common import load_config
from synth import Synth
c = load_config("experiments/20-scaling/016-mixed-es-en/config.json")
S = Synth(c, run="runs/esen/ttl.pt")
zref, rmask = S.style("../../voices/arf_06136.pt")   # voicepack, or a wav path for a new speaker
for i, t in enumerate(["Primera oración.", "Second sentence."]):
    wav, dur = S(t, zref, rmask, steps=16, cfg=2.0, duration_scale=1.0)   # lang="es" / "en" to force
    sf.write(f"out{i}.wav", wav, c["data"]["sample_rate"])

License

Weights and voicepacks CC BY-SA 4.0: training data is OpenSLR 61 (Google, CC BY-SA 4.0, https://www.openslr.org/61/) and LibriTTS-R (CC BY 4.0). The audit is in the repo's docs/LICENSES.md. Code MIT.