dots.tts.edit

dots.tts.edit

dots.tts.edit is a continuous autoregressive model for precise, instruction-controlled speech editing and zero-shot text-to-speech synthesis. It supports text replacement, insertion and deletion, emotion and prosody control, pauses, enhancement, and background-audio operations while preserving the speaker and the acoustic context outside edited regions.

• Code
• Paper
• Interactive Playground
• Demo Page

Model details

The model supports English and Mandarin speech editing and zero-shot speech synthesis. It produces 48 kHz audio. The core weights use BF16; the speaker encoder and vocoder retain FP32 weights.

Supported editing operations include:

• Text replacement, insertion, and deletion
• Localized emotion control
• Pitch and speaking-rate control
• Pause insertion and reduction
• Speech enhancement and background-audio operations
• Compositions of multiple edit operations

Installation

Install the latest main branch of dots.tts and its recommended dependencies:

git clone --branch main --single-branch https://github.com/studio-dots-ai/dots.tts.git
cd dots.tts
python -m pip install -e . -c constraints/recommended.txt

Speech editing CLI

Source audio, a tagged instruction, and the output path are required. Source and target transcripts are optional; when omitted, both are derived from the instruction.

dots.tts.edit \
  --model-name-or-path dots-studio/dots.tts.edit \
  --source-audio /path/to/source.wav \
  --instruction 'Hello <sub targ="small">brave</sub> world.' \
  --output edited.wav

Use --source-text and --target-text to override the derived transcripts. Source-speaker guidance defaults to auto: it is disabled when the instruction contains at least one operation and every operation is emo, bg, or enhance, and enabled for text, pitch, rate, pause, speaker-transfer, or mixed edits. Pass bare --use-xvector (or --use-xvector on) to force it on, and --use-xvector off to force it off. Speaker guidance uses a default scale of 1.5; override it with --speaker-scale when needed.

Python API

import soundfile as sf

from dots_tts.edit_runtime import DotsTtsEditRuntime

runtime = DotsTtsEditRuntime.from_pretrained(
    "dots-studio/dots.tts.edit",
    precision="bfloat16",
)
result = runtime.generate_edit(
    source_audio_path="/path/to/source.wav",
    instruction='Hello <sub targ="small">brave</sub> world.',
    # source_text and target_text are optional overrides.
    # use_xvector defaults to "auto"; pass True or False to override it.
)
sf.write(
    "edited.wav",
    result["audio"].float().cpu().squeeze().numpy(),
    result["sample_rate"],
)

Supported structural tags include <del>, <ins>, <sub targ="replacement">, <emo>, <pitch>, <rate>, <enhance>, <bg>, <pause/>, and <spk_transfer/>. Malformed instructions and instructions that derive an empty source or target transcript are rejected.

DotsTtsEditRuntime also provides generate and generate_stream for zero-shot TTS with the same checkpoint. See the code repository for examples.

Local Edit Playground

Build the committed frontend source with Node.js 20+ and launch the local app:

cd apps/edit_playground/frontend
npm ci
npm run build
cd ../../..
python apps/edit_playground/app.py \
  --model-name-or-path dots-studio/dots.tts.edit \
  --optimize

Limitations and responsible use

Generated or edited speech may contain pronunciation, timing, speaker-similarity, or instruction-following errors. Do not use the model to impersonate people, mislead listeners, violate consent, or create unlawful content. Users are responsible for complying with applicable laws and for clearly disclosing synthetic or edited audio where appropriate.

License

See the repository license metadata and the dots.tts project for the terms that apply to the weights and code.