kokoro-inno-clone-tuner

kokoro-inno-clone-tuner

pip install inno-kokoro

Zero-shot voice tuner for Kokoro-82M.

Outputs base Kokoro compatible voice packs @ [510, 1, 256]. (write-up)

Samples

Same passage for every voice, enrolled from the references. LibriTTS-R speakers are dev-clean held out from training.

Usage

Code available on Github: remsky/inno-kokoro.

Integrated into Kokoro-FastAPI (v0.9.0+)

pip install inno-kokoro

Enrolling a voice:

from inno_kokoro.enroll import Tuner, enroll, read
from kokoro import KPipeline

tuner = Tuner()  # pulls model.safetensors into the huggingface cache
pack, _ = enroll(*read("my_ref.wav"), tuner)

pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio

The pack is a plain tensor; torch.save(pack, "voices/am_me.pt") makes it a voice file like any other, prefixed by accent and gender like the stock packs.

The pitch-tracking ceiling is set automatically from the reference's harmonic spacing, so band-limited or archival sources land in the right octave without tuning.

• enroll(..., fmax=180) overrides it if a voice still reads the wrong register.

Architecture

A 5 to 30 s reference clip in, a stock-shaped Kokoro voice pack out. Everything loads from model.safetensors, no extra downloads.

<figure> <img src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/XKJQUMieuxKyu5xKu9dc3.png" alt="Inno-Kokoro enrollment architecture" width="95%" /> <figcaption>Enrollment Architecture Diagram: reference audio to Kokoro-compatible voice pack.</figcaption> </figure> <details> <summary><b>Architecture details</b></summary>

Enrollment embeds an input audio sample via the baked speaker encoder and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half.

• speaker encoder: a 6.6M-param WeSpeaker ResNet34 distilled from UniSpeech-SAT-sv embeddings
  • matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
• style head: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
• prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
  • Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
  • Packs more than 4 st from the reference's pitch are excluded.
  • Higher graded packs are weighted as preferred during selection.
• prosody head (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
  • fitted to deltas optimized through the frozen predictor. enroll(..., head=False) skips it.

Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.

Total model size @ fp16: ~24mb

</details>

Limits

• English trained (LibriTTS-R) currently
• Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
• Rythym is a best-effort match to the speaker.
  • Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
• Reference clip requirements:
  • 3 second minimum, up to a max of 30s
  • Reasonably free of audio artifacts
  • Single speaker

Training and Metrics

All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.

Trained via HF Jobs a10g-small; under $20 in GPU time:

<details> <summary>v0.1.0 training notes</summary>

• Stage 1: mapped speaker embedding to timbre:
  • Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
  • [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
• Stage 2: the embedding alone missed texture and pitch; adjustments:
  • Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
  • [11k steps: dev cosine 0.934, DNSMOS 3.30]
• Stage 3: tuned the encoder side to match the blended predictors:
  • Predictor half fixed to the blended rows, only the decoder half trained.
  • [3k steps: dev cosine 0.932, DNSMOS 3.48]
• Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
  • WeSpeaker ResNet34 plus a projection, trained on 1 - cos(student, teacher) over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
  • [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
• Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
  • Mixed real public-domain archival broadcast audio into the distillation crops.
  • [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]

</details> <details> <summary>v0.2.0 training notes</summary>

• Stage 1: the blend under-predicted pitch range and sentence-final falls:
  • Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
  • Targeted F0 mean, spread, IQR, pause fraction and speech time.
  • Ridge head from reference F0 mean + spread to the fitted delta.
  • [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
• Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
  • Added VoxPopuli and Emilia speakers to the distillation crops.
  • [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
• Stage 3: the timbre half had only seen LibriTTS speakers:
  • Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
  • [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]

</details>

Zero-shot identity benchmark

Inno gets about a third of the normalized similarity range, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).

<figure> <img src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/FmO3P3W8k501WZgqUBuyM.webp" alt="Zero-shot identity benchmark" width="95%" style="border: 1px solid black;" /> <figcaption> Benchmark Plot: Zero-shot speaker similarity comparison on LibriSpeech test-clean. </figcaption> </figure> <details> <summary><b>Full benchmark results and methodology</b></summary>

LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers. Reference in, new sentence out, scored against the speaker's real recording.

• normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
• RTF: RTX 4060 Ti

StyleTTS2 uses a related architecture with an unfrozen decoder, a possible ceiling for this style space.

• StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
• OpenVoice v2: MeloTTS EN-US, tau 0.3.
• F5-TTS: 32 NFE, CFG 2.

</details> <details> <summary><b>v0.1 vs v0.2 shipped-weight comparison</b></summary>

Measured on the shipped weights through the stock KPipeline: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.

The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.

</details>

Licenses and credits

Adapter weights and code: Apache-2.0, except the baked speaker encoder (enc.* in model.safetensors), which is CC BY-SA 3.0 (see below)

• Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture)
• The speaker encoder is a WeSpeaker ResNet34 (code Apache-2.0)
  • initialised from the VoxCeleb-trained ResNet34-LM weights (CC BY 4.0)
  • distilled from microsoft/unispeech-sat-base-plus-sv
    • (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub); encoder weights under CC BY-SA 3.0
• Training data (no audio redistributed beyond the samples):
  • LibriTTS-R (CC BY 4.0), also the LibriTTS sample references
  • VoxPopuli (CC0)
  • Emilia-YODAS EN (CC BY 4.0)
• Sample references:
  • Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022
  • Jane Goodall from BBC Woman's Hour excerpt, BBC, CC BY 3.0
  • Vincent Price public domain from "The House On Haunted Hill"

Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.