OpenVoice-ONNX-v2

OpenVoice V2 ONNX Models

This repository provides the speaker encoder and tone color converter from OpenVoice V2 in ONNX format for local voice cloning and voice conversion. The models can run through ONNX Runtime in C#, C++, Python, Rust, and other supported environments without a PyTorch runtime.

The original architecture and model weights were developed by MyShell and the OpenVoice contributors. See the upstream OpenVoice project.

The converter takes speech audio and changes its voice characteristics using source and target speaker embeddings. A separate text-to-speech system supplies the base speech when generating audio from text.

📦 Repository Contents

Download files using:

https://huggingface.co/Hinotsuba/OpenVoice-ONNX-v2/resolve/main/{filename}

Use a specific repository revision in place of main when pinning a deployment.

🛠️ Tensor Specifications

All floating-point tensors use float32. Both graphs have a fixed batch size of 1. frames denotes the number of spectrogram frames, not waveform samples.

Speaker Encoder: tone_extract.onnx

Reshape the extracted embedding to [1, 256, 1] before passing it to the converter.

Tone Color Converter: tone_color.onnx

For a single unpadded input, set audio_length[0] to frames. Read converted_audio[0, 0, :] to obtain the one-dimensional waveform. These samples are audio data; encoding them as WAV or another format is the caller's responsibility.

Audio Preparation

Prepare the spectrogram outside the ONNX graphs. The same preprocessing applies to reference audio and source audio; only the tensor axis order differs.

Decode and resample to mono float PCM at 22050 Hz, using the conventional PCM amplitude scale of approximately [-1, 1]. Then match the upstream spectrogram_torch implementation:

Use a library with these settings or reproduce the same window, padding, FFT scaling, and magnitude calculation. The graphs expect this linear spectrum, with no mel projection or logarithmic scaling.

🚀 Inference Workflow

1. Obtain clean reference speech from the source voice and the target speaker. For a TTS source, use a reference clip from the same base voice that will generate the speech.
2. Prepare each reference spectrogram and run tone_extract.onnx. Reshape the results to [1, 256, 1] for src_tone and dest_tone.
3. Generate or load the source speech. Prepare its spectrogram as [1, 513, frames].
4. Run tone_color.onnx with the source spectrogram, its frame count, both embeddings, and an explicit tau value.
5. Read the mono waveform from converted_audio and encode or play it at 22050 Hz. Resample separately if the output device needs another rate.

These models are used by Tsubaki TTS Engine for local C#/.NET text-to-speech with Piper synthesis, OpenVoice V2 voice cloning, and audio streaming without a Python runtime.

Export Behavior and Limits

Temperature: tau

tau is a required ONNX input and has no automatic default. In the upstream Python implementation, the internal voice_conversion() method defaults to 1.0, while the public ToneColorConverter.convert() method defaults to 0.3.

This export contains a fixed noise tensor. tau scales the latent perturbation drawn from that tensor: lower values reduce the perturbation, and 0 removes its contribution. Repeated CPU calls with identical inputs and tau values produced identical output in verification. Fresh noise is not sampled on each call.

Input Length

The converter's fixed noise tensor has shape [1, 192, 32768], limiting a single conversion to 32768 spectrogram frames despite the dynamic time axis. At a hop length of 256 and output rate of 22050 Hz, this corresponds to approximately 380 seconds of output audio. Split longer audio into shorter chunks before conversion.

Runtime and Export Metadata

CPU inference was verified with ONNX Runtime 1.29.0, including dynamic frame counts and explicit tau values. Execution-provider support and performance depend on the runtime, hardware, and application.

The export script and exact upstream checkpoint revision are not included in this package. The metadata above describes the distributed ONNX files.

⚖️ License and Attribution

The original OpenVoice V2 architecture and weights are released by MyShell and the OpenVoice contributors under the MIT License. The ONNX model files in this repository are distributed under the same terms.

Commercial and non-commercial use are permitted under the MIT License. Preserve the copyright and permission notice when redistributing the files. See LICENSE for the full original notice, including Copyright 2024 MyShell.ai.

Upstream project: myshell-ai/OpenVoice.