CTC-TTS

CTC-TTS: LLM-Based Dual-Streaming Text-to-Speech with CTC Alignment

Overview

This repository contains official pre-trained checkpoints for CTC-TTS, proposed in our Interspeech 2026 paper CTC-TTS: LLM-Based Dual-Streaming Text-to-Speech with CTC Alignment.

CTC-TTS is a LLM-driven dual-streaming TTS framework, designed to solve two critical pain points of existing streaming synthesis systems:

1. Heavy pipeline dependency on MFA (Montreal Forced Aligner) for text-speech alignment
2. Fixed-ratio token interleaving that fails to model real phoneme-speech alignment rules

We replace MFA with a CTC-based neural aligner (built on Whistle ASR model from THU-SPMI Lab) and propose a bi-word interleaving block structure. Two variants are provided for flexible quality-latency tradeoff:

• CTC-TTS-L: Concatenate text & speech tokens along sequence dimension, higher synthesis fidelity, slightly larger first-packet latency
• CTC-TTS-F: Stack phoneme/speech embeddings along feature dimension, ultra-low streaming latency for real-time scenarios

Experiments on single-speaker streaming and multi-speaker zero-shot synthesis show consistent improvements over LLMVox, ELLA-V and other MFA-aligned baselines.

Key Contributions

1. Lightweight CTC phoneme-speech alignment pipeline, eliminating MFA offline forced alignment
2. Bi-word context-aware interleaving strategy with compact look-ahead window
3. Dual model variants (L/F) to balance speech naturalness and streaming latency

Pre-trained Checkpoints

Experimental Results

1. Single-Speaker Streaming Benchmark (VoiceAssistant400K, vs LLMVox)

Greedy search. FPL-A = first-packet latency assuming full text is available.

• CTC-TTS-F achieves lower WER/CER and shorter FPL-A than LLMVox, demonstrating the advantage of CTC alignment + bi-word sequences over fixed-ratio interleaving.
• CTC-TTS-L delivers even lower WER/CER at the cost of slightly higher latency.

2. Multi-Speaker Zero-Shot — Continuation Task (LibriSpeech test-clean)

Given a text segment and its corresponding 3-second prefixed speech, synthesize speech for the remaining text. Nucleus sampling.

• CTC-TTS-L outperforms MFA+bi-word in all metrics except SPK, validating the superiority of CTC alignment over MFA.
• Our methods outperform CTC+ELLA-V across the board, confirming the advantage of bi-word interleaving over ELLA-V's local-advance sequence.

3. Multi-Speaker Zero-Shot — Cross-Speaker Task (Seed-TTS test-en)

Given ~3 seconds of speech and its transcribed text as a prompt, synthesize speech for another utterance (out-of-domain). Nucleus sampling.

• CTC-TTS-L achieves near-optimal performance on this out-of-domain task.
• CTC+ELLA-V outperforms MFA+ELLA-V in WER/CER, indicating CTC alignment generalizes better to out-of-domain cross-speaker scenarios than MFA.
• ELLA-V's sequence organization yields poor performance in both alignment settings, highlighting the benefit of bi-word interleaving.

Evaluation Metrics

• Objective: WER / CER (intelligibility), UTMOS (naturalness), SPK (speaker similarity via WavLM-Base-Plus-SV), FPL-A (first-packet latency)
• Subjective: MOS (naturalness) and SMOS (speaker similarity), 30 samples per system, 20 listeners, 1–5 scale with 95% confidence intervals
• ASR backbones: whisper-large-v3 (single-speaker), Conformer-Transducer xlarge (multi-speaker)

Citation

If you use this model or method in your research, please cite our paper:

@article{liu2026ctc,
  title={CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment},
  author={Liu, Hanwen and Yusuyin, Saierdaer and Huang, Hao and Ou, Zhijian},
  journal={arXiv preprint arXiv:2602.19574},
  year={2026}
}