gemma1b-tts-integration

Route-2 v12 — phoneme-conditioned flow-matching TTS (PT-BR)

Phoneme rebuild of Route 2. Pipeline: text → espeak-ng pt-br G2P → phoneme IDs → nn.Embedding → Matcha OT-CFM (duration predictor + MAS) → mel → frozen Vocos → 24 kHz. Lip-sync from the duration predictor's w_ceil @ 93.75 Hz.

Production gate: ASR (whisper PT) text_similarity ≥ 0.55 (0.41 = prior AR plateau).

Status (2026-05-29)

• Smoke PASSED (smoke_v12.py): G2P, Embedding wiring, padding_idx zero, loss falls, lip-sync invariant sum(w_ceil)==y_length.
• Overfit PASSED (overfit_v12.py): 16 real utts, sim_mean 0.7154, median 0.787, 4 exact. Phoneme conditioning works; v11's failure was conditioning granularity, not Matcha/Vocos.
• Full train: code ready; not yet run (blocked on stable GPU pod availability).

Files

• r2_model_v12.py — TucanoMatchaTTS, Embedding in_proj (in_dim = phoneme vocab). 19.05M params.
• phon_util.py — espeak-ng pt-br G2P (punctuation-glue fixed). phonemize_batch / build_vocab / encode.
• gen_mel.py — Mimi decode (codes→audio) → Vocos mel targets. Out: <pref>_concat.npy + _lengths.npy + _ids.json. Mimi via MIMI_PATH env or kyutai/mimi.
• prep_phon_v12.py — 94k jsonl → phon_vocab.json + r2_phon_ids.json (per-id ID lists).
• r2_train_v12.py — phoneme loader + OT-CFM train (dur+prior+diff), AdamW, ckpt full-state (model+opt) + --resume.
• pod_pipeline.sh — on-pod: setup → pull dataset (R2) → gen_mel → prep_phon → train. Needs RCLONE_CONFIG_R2_* env.
• smoke_v12.py, overfit_v12.py — validation tests.
• route2_nova_arquitetura.html — architecture explainer + APA references.

Data

94k utts / ~106 h PT-BR in R2 bucket tts-ptbr-training/data/ (train_94k.jsonl + val.jsonl). Keys: speakable_text (G2P), mimi_codes_flat (→mel), duration_frames, id.

Run (on a GPU pod)

# scp these scripts to the pod, set RCLONE_CONFIG_R2_* env, then:
STEPS=30000 BATCH=32 bash pod_pipeline.sh   # setup -> data -> gen_mel -> prep -> train