dacvae-tts-tr-w512

DACVAE-TTS Turkish (width 512, 66.5M)

Zero-shot voice-cloning text-to-speech for Turkish, trained from scratch with dacvae-tts in the latent space of the frozen Meta DACVAE codec (facebook/dacvae-watermarked, 48 kHz, 128 channels, 25 latent frames/s). Text in (UTF-8 bytes, Turkish number normalization), 3–15 s reference audio + its transcript in, 48 kHz speech in the reference voice out.

• Architecture: 12 DiT blocks, width 512, 8 heads, single-frame latent tokens, RoPE self-attention, length-aware RoPE text cross-attention, QK-norm, shared low-rank AdaLN (rank 64), byte text encoder (4 conv + 4 self-attention blocks), EDM-style flow-matching target, auxiliary CTC head, skip/repeat contrastive negatives, Muon optimizer, EMA 0.9999.
• Training data: Vyvo/tr-dataset-12 (Turkish podcast segments, 93 h) filtered to quality ≥ 55, Whisper-large-v3 CER ≤ 0.10 and DNSMOS OVRL ≥ 2.8 → 32k clips (~70 h). 60k updates, batch expansion 4, one RTX 4090 (9.5 h).
• Checkpoint selection: by Freya-TR-Eval only (495 conversational sentences that are not in the training data, synthesized with 24 held-out speaker prompts), to exclude checkpoints that memorize the training set. This checkpoint (tr-w512-clean, update 60k) had the best Freya score of all runs; an in-domain fine-tune that looked better on the training-domain monitor regressed on Freya and was not published.

Results (Freya-TR-Eval, faster-whisper large-v3 beam 5, Turkish normalization, guidance 5, 32 Euler steps)

Also: speaker similarity 0.946 (WavLM-base-plus-SV cosine vs the prompt), DNSMOS OVRL 2.89 (real podcast speech through the codec: 3.27). Held-out-speaker cross-utterance cases (48): WER 0.099 / CER 0.053 at guidance 5. Generated audio for every evaluated checkpoint, with texts and transcripts, is in VoiceHub/dacvae-tts-tr-w512-clean (dataset).

Recommended inference settings (validated on Freya-TR-Eval)

The numbers above use the plain prompt-rate duration rule and one sample per sentence. Inference-side improvements, each measured on Freya-TR-Eval with two disjoint draws of the 24 unseen prompt voices (prompt seed 42 = the set of the table above, seed 1000 = a second set):

Guidance variants did not beat plain CFG on intelligibility: APG (η 0.5, momentum −0.3) and CFG-rescale (φ 0.7) remove most of the output saturation (files reaching the decoder's tanh ceiling 96% → 58% / 22%, loudness −13.9 → −16.3 / −18.4 LUFS) and raise speaker similarity/DNSMOS slightly, but WER rises to 4.96% / 5.16%. Independent text/speaker guidance (speaker 3 or 7 with text 5) and interval CFG (t < 0.7) did not help either (WER 4.63 / 5.14 / 4.42). Best-of-3 is ranked by Whisper-large-v3-turbo and scored by Whisper-large-v3; turbo is distilled from large-v3, so part of the gain may reflect shared ASR preferences. Speaker similarity (0.947) and DNSMOS (2.93) are unchanged by the recommended settings. All runs, per-sentence transcripts and scores: VoiceHub/dacvae-tts-tr-w512-clean, folder demo-experiments/.

Usage

pip install torch torchaudio numpy soundfile scipy pyloudnorm huggingface_hub transformers \
  "dacvae @ git+https://github.com/facebookresearch/dacvae.git@414c20785fc3a28373073ea8ef7a1316eeeaca6e"

import sys
from huggingface_hub import snapshot_download

repo = snapshot_download("VoiceHub/dacvae-tts-tr-w512")
sys.path.insert(0, repo)                      # the `dacvae_tts` package (Turkish frontend, duration modes) is bundled here
from dacvae_tts.frontend import speakable, split_sentences
from dacvae_tts.inference import Synthesizer

tts = Synthesizer(f"{repo}/model.pt", device="cuda")   # downloads facebook/dacvae-watermarked on first use
voice = tts.prepare_reference("prompt.wav", "Prompt kaydında söylenen cümlenin tam metni.")  # 3-15 s, one speaker
text, changes = speakable("Toplantı 23.09.2026'da saat 14:30'da; bütçe %20 arttı.")  # dates, times, symbols -> words
results, info = tts.synthesize_many(
    [chunk for chunk, _ in split_sentences(text)], voice, candidates=3,   # sentence chunks x candidates, one batch
    duration_mode="auto", guidance=5.0, steps=32, sway=-1.0,
)
# results[i][k]["audio"] is candidate k of chunk i (48 kHz float32); rank the candidates with an ASR (the demo uses
# openai/whisper-large-v3-turbo and keeps the lowest CER), then join the chunks.

Synthesizer.synthesize(text, ref_audio=..., reference_text=..., duration_mode="auto", guidance=5.0, steps=32) is the one-call API for a single sentence. Without reference_text the prompt is transcribed with Whisper (asr_language="tr"). Sampler options: guidance_until/guidance_from (interval CFG), cfg_rescale, apg_eta/apg_norm/apg_momentum, speaker_guidance (independent text/speaker guidance).

Limitations

• 48 kHz output, but the training audio is podcast MP3 (mostly 12–16 kHz bandwidth); DNSMOS ≈ 2.9.
• Remaining errors: occasional repeated or swapped words (best-of-N removes most of them), letter errors in rare foreign names; quality depends on the prompt (per-voice Freya WER 0.6–9.6 %; fast and noisy prompts are the hardest).
• High guidance (5) saturates the decoder output (loudness ~−14 LUFS, samples at the tanh ceiling in most files); the demo normalizes to −16 LUFS; APG avoids the saturation at some cost in WER.
• Reference audio needs its transcript (typed or via Whisper); reference and target should be different sentences.
• Non-commercial: the training data (Vyvo/tr-dataset-12, CC-BY-4.0 podcasts) and this model are released CC-BY-NC-4.0; the DACVAE codec keeps its upstream license.

Demo Space

Live demo (ZeroGPU): https://huggingface.co/spaces/Vyvo/dacvae-tts-tr-demo

• Synthesis with the settings above (automatic speaking rate, best-of-3, long text split into sentences, numbers/dates/ currencies/abbreviations read out), Whisper transcript of the reference filled in and editable, WER/CER, speaker similarity and DNSMOS for every output, −16 LUFS 48 kHz WAV.
• Research tools: any checkpoint from the Hub (log in for private repositories — your own permission is used), A/B comparison of 2–4 checkpoints, batch test on sentence lists or Freya-TR-Eval with a zip of all outputs, API access (gradio_client, endpoint /synthesize).

The demo's code is in space/ (app.py interface, engine.py models and ZeroGPU jobs; the dacvae_tts package at the repository root is reused); space/SPACE.md describes deploying a copy.

Files

• model.pt: EMA weights + config + codec metadata + latent normalization (torch.load(..., weights_only=True)).
• config.json: training/model configuration.
• dacvae_tts/: inference code (snapshot of the training repository, branch turkish-tts): Turkish text frontend (frontend.py), duration rules and the fitted predictor (duration.py, duration_tr.json), sampler options, batched synthesis (inference.py), output post-processing (audio.py).
• space/: the Gradio demo.