kokoro-ninja

Kokoro Ninja 🥷 — fast multilingual zero-shot voice cloning

Clone a Vietnamese voice from a few seconds of audio at 18× realtime on a single GPU. It also speaks English, Chinese, French, German, Japanese and Korean — but cloning fidelity outside Vietnamese is measurably weaker, and the numbers below say so rather than claiming seven equal languages.

StyleTTS2 + a CAMPPlus speaker encoder. One non-autoregressive forward pass: no diffusion sampler, no autoregressive decode loop. Generation is deterministic.

Code: https://github.com/primepake/kokoro-ninja
Demo: https://primepake.github.io/kokoro-ninja/

Usage

import soundfile as sf
from kokoro_ninja import KokoroNinja

tts = KokoroNinja.load(
    checkpoint_path="kokoro-ninja-multilingual.pth",
    config_path="config_inference.yml",
    campplus_onnx_path="campplus/campplus.onnx",
    device="cuda:0",
)

style = tts.compute_style("reference.wav")     # 3-10s of the voice to clone

for lang, text in [
    ("vi", "Xin chào, đây là giọng nói được nhân bản."),
    ("en", "Hello, this is a cloned voice."),
    ("ja", "こんにちは、これはクローンされた声です。"),
]:
    wav, sr = tts.synthesize(text=text, ref_s=style, language=lang)
    sf.write(f"out_{lang}.wav", wav, sr)

One reference clip carries the voice across every supported language. compute_style() is per-voice — compute once, reuse for every sentence.

Requires espeak-ng (the phonemizer backend). Easiest correct install: pip install espeakng-loader, which bundles 1.52 with complete dictionaries. The model was trained against 1.50, but we measured 1.50 vs 1.52 as producing identical phonemes on all 30 Vietnamese benchmark sentences and on en/fr/de/ja/ko.

If you build espeak from source, compile the dictionaries explicitly — a corrupt en_dict makes espeak spell words out letter by letter ("B-O-N-J-O-U-R"), silently wrecking English, French and German. See the repo README.

Files

Benchmarks

One NVIDIA L4, 105 held-out speakers, identical references and sentences for every system. References are 3–10 s in both languages so they are comparable.

Vietnamese — all 65 VIVOS speakers, 130 utterances

7.3× faster than omnivoice-vietnamese, 13.4× faster than VieNeu v3 Turbo on the same GPU, and we edge out VieNeu Turbo on both intelligibility and naturalness. We lose on speaker similarity: 87% of the human ceiling versus their 99%. clone% rescales SECS between the cross-speaker floor (0.725 — two different speakers) and the human ceiling (0.936).

English — 40 LibriSpeech test-clean speakers, 80 utterances

English does not clone well. Output is fast and broadly intelligible, but 9% of the usable similarity range (floor 0.681, ceiling 0.960) is barely above "different speaker" — the model speaks English, it does not reliably carry a voice into it. This is not an artifact of the eval: matching the English reference-duration window to Vietnamese moved SECS by 0.001. Treat the six non-Vietnamese languages as speech synthesis, not voice cloning, until this improves.

RTF is full-utterance compute ÷ audio duration, not time-to-first-audio. SECS uses WavLM-base-plus-sv rather than CAMPPlus, since scoring with the encoder the model is conditioned on would flatter it. WER uses PhoWhisper for Vietnamese and Whisper for English. UTMOSv2 is nondeterministic (up to 0.08 between runs on identical audio) — treat small MOS gaps as noise; SECS and WER are exactly reproducible.

Always end your text with punctuation

The model is trained on sentence-final punctuation and is sensitive to it: text without a terminal mark tends to clip or run on at the end. synthesize() appends one when your text lacks it. Adding the missing period moved WER from 5.1% to 3.9% and MOS from 2.47 to 2.66 on an earlier run of this benchmark.

Training data

Base checkpoint: StyleTTS2's LibriTTS model, adapted to the Vietnamese phoneme vocabulary and extended to the multilingual token set.

Licence — non-commercial

CC-BY-NC-SA-4.0. This is inherited, not chosen: viVoice is CC-BY-NC-SA (NonCommercial and ShareAlike) and the Emilia subset is CC-BY-NC, so the strictest terms propagate to anything trained on them.

You may use these weights for research, evaluation and personal projects. You may not use them in a commercial product, and derivatives must carry the same licence.

The inference code is separately Apache-2.0.

Limitations

• Speaker similarity trails the best autoregressive/codec models (see benchmarks).
• No text normalization — expand numbers, dates and currency before synthesis. This is the most common source of real-world errors and is not handled here.
• Noisy or very short (<3 s) references degrade cloning; there is no built-in denoiser.
• Cloning fidelity outside Vietnamese is weak (see the English table). Chinese, French, German, Japanese and Korean are unmeasured; expect the same pattern.
• Japanese input must be kana — espeak-ng reads hiragana and katakana but silently drops kanji, so convert with pyopenjtalk/MeCab first. Every other language takes normal text.
• 24 kHz output, no streaming.

Acknowledgements

StyleTTS2 (Li et al.), 3D-Speaker / CAMPPlus, VIVOS for evaluation, PhoWhisper (VinAI) and UTMOSv2 for scoring.