dubclean-voice

DubClean Voice

Reference-conditioned removal of the original-language speech from a dubbed soundtrack.

Russian voice-over releases usually keep the English dialogue audible under the translation. DubClean Voice takes the speech of the dubbed track together with the speech of the original track — the reference — and returns the translation with the matched foreign speech removed.

The model is the speech-cleaning stage of DubClean, a local Windows service for rebuilding a dubbed soundtrack. It is published on its own so that the checkpoint can be inspected, reused and evaluated independently of that application.

Files

Both checkpoints hold bit-identical weights and produce identical output. The production file additionally carries the AdamW optimizer state and the training history of the run that produced it, and its SHA-256 is the one recorded in DubClean Portable's portable_manifest.json. For plain inference prefer dubclean_voice_inference.pt.

Quick start

pip install torch numpy scipy soundfile huggingface_hub
hf download Yaromanol/dubclean-voice --local-dir dubclean-voice
cd dubclean-voice

python selftest.py --checkpoint dubclean_voice_inference.pt

python infer.py \
  --checkpoint dubclean_voice_inference.pt \
  --mixture dubbed_speech.flac \
  --reference original_speech.flac \
  --output translated_speech_clean.flac

From Python:

import soundfile as sf
from dubclean_voice import load_model
from infer import read_mono, separate

model = load_model("dubclean_voice_inference.pt", device="cpu")
mixture = read_mono("dubbed_speech.flac")     # translated speech + leaked foreign speech
reference = read_mono("original_speech.flac") # foreign speech to remove
voice = separate(model, mixture, reference, device="cpu")
sf.write("translated_speech_clean.flac", voice, 24000)

The model itself is a plain nn.Module:

output = model(mixture_tensor, reference_tensor)  # float32 [B, T] at 24 kHz
output["voice"]      # translated speech
output["reference"]  # removed component = mixture - voice

What the inputs must be

This is the part that decides whether the model works at all.

1. Speech stems, not raw film audio. Both inputs are the output of a speech extractor run over the two soundtracks — in DubClean, MossFormer2_SE_48K (Apache-2.0) over the original track and a DubClean-tuned variant over the dubbed track. Music and effects are removed beforehand and restored later. Feeding a full mix is out of domain.
2. Time-aligned. The network compares the two signals frame by frame; the training data covers shifts up to about 150 ms and mild speed edits, and the dilated bottleneck gives some tolerance beyond that, but a systematic offset or a drifting timeline must be corrected first. DubClean aligns the two tracks (correlation and per-scene offsets) before this stage.
3. The same scene. Mixture and reference are truncated to the shorter of the two. Passing unrelated audio as the reference is a trained-for negative case: the model is supposed to leave the mixture alone, not to invent removals.
4. 24 kHz mono. infer.py resamples and downmixes for you. The output is band-limited to 12 kHz — inside DubClean the full-band music and effects come from a separate model, so the band limit applies only to the speech layer.

Architecture

STFT with n_fft=512, hop_length=256, Hann window, center=True → 257 bins at 24 kHz. Six input planes are stacked from the mixture spectrum X and the reference spectrum R:

• log1p(12·|X|) and log1p(12·|R|);
• a time-averaged reference profile (long-term spectral shape of the reference);
• the spectral flux of the mixture;
• the log-magnitude difference;
• a normalized magnitude similarity between mixture and reference.

Phase relation never enters the network. A U-Net (base 16 channels → 16/32/64/96, bottleneck 128, skip connections) processes those planes; the bottleneck holds ten identity-initialised residual blocks with time-axis dilations 1,2,4,8,16,1,2,4,8,16, which is what carries tolerance to jitter and local timing drift.

The two output planes form an unbounded complex ratio mask

mask = (1 + 2·tanh(a)) + i·(2·tanh(b))

applied to the mixture spectrum, followed by iSTFT. The mask is deliberately not constrained to 0…1: cancelling speech that overlaps in time and frequency needs gains above one, negative real parts and a real phase correction. A narrow mask capped suppression at roughly 4–6 dB in earlier versions of this model.

The removed component is defined as mixture − voice, so the two outputs always sum back to the input.

Inference is windowed: 6 s windows, 3 s hop, raised-cosine cross-fade. Memory use does not grow with file length. On CPU (torch 2.5.1, Windows 10) the reference implementation runs at roughly 8× real time; a GPU is faster but not required.

Training data

The model was trained on semantic-balanced-replay-20260720-v1: 98,242 examples of 6.0 s at 24 kHz, split 82,241 / 8,001 / 8,000 into train / validation / test.

Every example is synthesized from real speech clips: a translated-speech target, a foreign-speech component embedded under it, and the reference the model is conditioned on, mixed under a controlled factor plan (18 factors, 113 options — voice types, overlap, ducking, reference shift, speed edits, level ratios, backgrounds, extractor quality). Because the mixture is composed explicitly, the ground-truth target and the ground-truth removed component are exact.

Composition by supervision policy:

20% of the training split are negative cases (no_op, preserve_all_speech, preserve_ru), replayed on purpose: a separator that also has to not act is much safer on real films than one trained only on removals.

21 scenarios are represented, among them typical voice-over, monologue, dense dialogue, equal-level overlap, foreign speech louder than the translation, very low embedded foreign speech, partial or extra-phrase references, wrong reference, timing drift, local edit jumps, similar-sounding voices, speech-like background, and a 32,000-example "hard master" block in which the two tracks are pushed through independent mastering chains (5.1→stereo downmix, AC-3→AAC, broadcast versus release master, piecewise gain envelopes, time-varying ducking, micro-timing jitter, codec+EQ+dynamics chains).

Source material: speech clips extracted from film soundtracks held by the maintainer — 12,959 clips (~25 h) from 45 audio tracks of 25 works, Russian and English. Clips are split by source work (16 works train, 3 validation, 4 test), so validation and test films never contribute speech to training.

Individual titles are not published. See TRAINING_DATA_DISCLOSURE.md: publishing this model grants no rights to the films, translations or recordings behind the dataset, and the dataset itself is not distributed.

Training procedure

• AdamW, weight decay 1e-5; the final continuation ran at lr 1e-5, batch 4 with 4 gradient-accumulation steps, on a single CUDA GPU.
• Loss: complex-spectrogram L1 on both outputs (1.0 / 0.6), waveform L1 (0.8 / 0.25), a small negative-SI-SDR term on the translated speech (0.002), a cosine leakage penalty against the removed component (0.10), and a residual energy term normalized by the foreign-speech energy (1.00). The last term is what makes "removed nothing" expensive when there is something to remove, while staying harmless on no-op cases.
• Early stopping on validation loss, patience 2. Epoch 37 (91,323 steps) was selected; epochs 38 and 39 were worse and training stopped.

Evaluation

Metrics on the held-out splits of the synthetic benchmark, at the selected epoch. EN suppression is 10·log10( MSE(mixture − target) / MSE(output − target) ) — how much of the error against the clean translation is removed. RU SI-SDR is the scale-invariant SDR of the output against the clean translation, i.e. how intact the translation stays.

By channel condition (validation, 40 examples each):

Acceptance run for the release, on top of the table above:

• hard-master bucket: 6.90 dB suppression at 13.83 dB RU SI-SDR;
• safety bucket (nothing to remove): 107.6 dB RU SI-SDR — on no-op material the model is effectively a passthrough;
• 25 real dubbed-film samples were rendered through the full application and reviewed by ear; no failures were recorded against the previous production checkpoint.

These numbers are a synthetic benchmark. Read them as a comparison between checkpoints, not as a promise about a particular film.

Limitations and failure modes

• A dub mastered from a different source can be unremovable. If the translated release was mixed from a different master than the original track you hold, the embedded foreign speech is not phase- or timbre-coherent with your reference, and suppression measured on real films can fall to ~0 dB with the model behaving correctly. This is a property of the material, not a bug — check a preview scene before processing a whole film.
• Alignment is a precondition. Uncorrected offsets beyond a few hundred milliseconds, edit differences or a drifting timeline break the frame-by-frame comparison.
• The extractor is part of the domain. The model was trained on extractor output, including its artifacts; with a different speech extractor, expect a domain shift, and note that the extractor itself can damage the translated speech before this model ever sees it.
• Language pair. Trained on Russian voice-over above English originals. Other pairs are untested.
• Bandwidth. 24 kHz internally; the speech layer comes back band-limited to 12 kHz.
• Not a standalone dub cleaner. Alone it processes speech stems. Producing a finished soundtrack additionally needs alignment, music/effects restoration and remixing — that is what the DubClean application does around it.
• Short events. The reported metrics come from long-form buckets; very short isolated utterances were not measured separately in this run.

Intended use

Restoring the speech layer of a dubbed soundtrack you have a lawful basis to process — personal archives, restoration work, dubbing and localisation research, and evaluation of reference-conditioned separation.

Out of scope: passing off the output as an official release, and any use of source material you have no right to process. The AGPL grants rights to this model and its code only. It grants nothing over films, translations, recordings or subtitles.

The DubClean pipeline

original track ──► speech extractor ──────────────► reference (EN speech)
                                                          │
dubbed track  ──► speech extractor ──► alignment ──► mixture (RU + leaked EN)
                                                          │
                                                          ▼
                                                  DubClean Voice  ← this model
                                                          │
                                       clean RU speech ───┼──► remix ──► soundtrack
                                                          │
              dubbed track ──► DubClean M&E ──► music and effects

DubClean M&E, the companion model that rebuilds speech-free music and effects, is part of the DubClean application and is not published here.

License and rights

The weights and the code in this repository are AGPL-3.0-or-later. The full text is in LICENSE; scope and exclusions are in MODEL_LICENSE.md; attribution and third-party notes are in NOTICE.

If you run a modified version of this model as a network service, the AGPL requires you to offer the corresponding source to its users. For terms outside the AGPL, see PROPRIETARY_LICENSING.md.

Media processed with DubClean is not licensed by DubClean, and processing it does not place it under the AGPL.

Citation

@software{dubclean_voice_2026,
  title  = {DubClean Voice: reference-conditioned removal of original-language
            speech from dubbed soundtracks},
  author = {DubClean contributors},
  year   = {2026},
  note   = {AGPL-3.0-or-later},
  url    = {https://huggingface.co/Yaromanol/dubclean-voice}
}

Russian version of this card: README_RU.md.