CtrlSpeech

CtrlSpeech

Checkpoints for CtrlSpeech — expressive speech synthesis with coarse-to-fine latent control. Code, CLI and interactive demo: https://github.com/zszheng147/CtrlSpeech

CtrlSpeech is a zero-shot TTS model you can steer after generating. It exposes the pitch contour, loudness contour and phoneme boundaries of any utterance, lets you edit one of them, and resynthesises following the edit while leaving the others alone. Prosody is conditioned per phoneme token, so a single word can be stretched while the rest of the sentence keeps its timing.

Contents

Each model folder holds model.safetensors, a training-free config.yaml, and the qwen_config.json describing the AR backbone architecture. The fine-tuned backbone weights are inside the checkpoint; no Qwen weights are redistributed and none are downloaded at load time.

Usage

pip install git+https://github.com/zszheng147/CtrlSpeech

from ctrlspeech import CtrlSpeech, shift_pitch_semitones

tts = CtrlSpeech.from_pretrained("control-600m")     # downloads on first use
baseline = tts.from_audio("clip.wav", "the transcript of that clip")
tts.regenerate(baseline, pitch=shift_pitch_semitones(baseline.gen_f0, 5)) \
   .save("higher.wav")

Duration editing and adopting your own recording require Montreal Forced Aligner with the english_us_arpa acoustic model and dictionary.

Control representation

Analysis runs at 100 frames/s (16 kHz, hop 160).

• Pitch — 128 mel-spaced bins; bin 0 is unvoiced.
• Loudness — 64 A-weighted dB bins, about 1.05 bins per dB.
• Duration — per-phoneme frame counts, at most 191 frames (1.91 s) per phoneme; an edited timeline is capped at 2001 frames (20 s).