kani-tts-2-pt

<p> <img src="public/logo.png" alt="Logo" width="100%"> </p>

KaniTTS2

Text-to-Speech Model with Frame-level Position Encodings, optimized for Realtime Conversations

Overview

KaniTTS2 uses a two-stage pipeline combining a LLM with an FSQ audio codec.

Key Specifications:

• Model Size: 400M parameters
• Sample Rate: 22kHz
• Languages: English, Spanish, Kyrgyz

Audio Examples

Installation

pip install kani-tts-2
pip install -U "transformers==4.56.0"

Quickstart

from kani_tts import KaniTTS

# Initialize model
model = KaniTTS('repo/model')

# Generate speech
audio, text = model("Hello, world!")

# Save to file
model.save_audio(audio, "output.wav")

Voice Cloning

from kani_tts import KaniTTS, SpeakerEmbedder

# Initialize models
model = KaniTTS('repo/model')
embedder = SpeakerEmbedder()

# Extract speaker embedding from reference audio
speaker_embedding = embedder.embed_audio_file("reference_voice.wav")

# Generate speech with cloned voice
audio, text = model(
    "This is a cloned voice speaking!",
    speaker_emb=speaker_embedding
)
model.save_audio(audio, "cloned_voice.wav")

For more examples including language tags, generation parameters, and the full API reference, see the kani-tts-2 package.

Performance

Nvidia RTX 5080 Benchmarks:

• RTF: ~0.2 (using custom executor)
• Memory: 3GB GPU VRAM

Pretraining:

• Dataset: ~10k hours
• Hardware: 8x H100 GPUs, 6 hours training.

Use Cases

• Conversational AI: Real-time speech for chatbots and virtual assistants
• Research: Fine-tuning for specific voices, accents, or emotions

Limitations

• Performance degrades with inputs exceeding ~40sec
• May inherit biases from training data in prosody or pronunciation
• Optimized primarily for English; other languages may require continual pretraining

Optimization Tips

• Multilingual Performance: Continually pretrain on target language datasets and fine-tune NanoCodec
• Batch Processing: Use batches of 8-16 for high-throughput scenarios
• Hardware: Optimized for NVIDIA Blackwell architecture GPUs

Resources

Models: Pretrained Model, English Model

Github: kani-tts-2 package.

Pretraining Framework Train your own TTS model on your language or accent from scratch using this open-source pretraining framework: KaniTTS2-Pretrain.

Example Dataset: https://huggingface.co/datasets/nineninesix/kanitts2-es-nano-codec-speaker-emb-dataset

Openai compatible streaming API example: https://github.com/nineninesix-ai/kani-tts-2-openai-server

Links: Website, Discord

vLLM Integration

Coming soon.

Tech Report

Coming soon.

Acknowledgments

Built on top of LiquidAI LFM2 350M as the backbone and Nvidia NanoCodec for audio processing.

Training data provided by the LAION team through the Emilia and EmoNet-Voice datasets.

Speaker embeddings extracted using Orange/Speaker-wavLM-tbr.

Citation

@article{liquidai2025lfm2,
 title={LFM2 Technical Report},
 author={Liquid AI},
 journal={arXiv preprint arXiv:2511.23404},
 year={2025}
}

@inproceedings{emilialarge,
  author={He, Haorui and Shang, Zengqiang and Wang, Chaoren and Li, Xuyuan and Gu, Yicheng and Hua, Hua and Liu, Liwei and Yang, Chen and Li, Jiaqi and Shi, Peiyang and Wang, Yuancheng and Chen, Kai and Zhang, Pengyuan and Wu, Zhizheng},
  title={Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation},
  booktitle={arXiv:2501.15907},
  year={2025}
}

@article{emonet_voice_2025,
  author={Schuhmann, Christoph and Kaczmarczyk, Robert and Rabby, Gollam and Friedrich, Felix and Kraus, Maurice and Nadi, Kourosh and Nguyen, Huu and Kersting, Kristian and Auer, Sören},
  title={EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection},
  journal={arXiv preprint arXiv:2506.09827},
  year={2025}
}

@inproceedings{gengembre24_interspeech,
  title     = {Disentangling prosody and timbre embeddings via voice conversion},
  author    = {Nicolas Gengembre and Olivier {Le Blouch} and Cédric Gendrot},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {2765--2769},
  doi       = {10.21437/Interspeech.2024-207},
  issn      = {2958-1796},
}

Responsible Use

Prohibited activities include:

• Illegal content or harmful, threatening, defamatory, or obscene material
• Hate speech, harassment, or incitement of violence
• Generating false or misleading information
• Impersonating individuals without consent
• Malicious activities such as spamming, phishing, or fraud

By using this model, you agree to comply with these restrictions and all applicable laws.

If you use this code in your research, please cite:

@software{kani_tts_2,
  author = {Nineninesix},
  title = {KaniTTS2: Text-to-Speech Model with Frame-level Position Encoding},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://github.com/nineninesix-ai/kani-tts-2}},
  note = {Open-source TTS model}
}

Contact

Have a question, feedback, or need support? Please fill out our contact form and we'll get back to you as soon as possible.