magpie_tts_multilingual_357m

MagpieTTS Multilingual 357M

<style> img { display: inline; } </style>

| |

🤗 HuggingFace MagpieTTS Multilingual demo: magpie_tts_multilingual_demo

💻 NeMo Framework: github.com/NVIDIA/NeMo

[!Note] March 3, 2026: MagpieTTS v2602 was released with support for 2 new languages (Hindi and Japanese).

For the older checkpoint released in January 2026, refer to the v2512 branch.

Description:

The model is a text-to-speech model that generates speech in 5 different English speakers - Sofia, Aria, Jason, Leo, John Van Stan. Each speakers can speak nine different languages (En, Es, De, Fr, Vi, It, Zh, Hi, Ja). The model predicts discrete audio codec tokens autoregressively using a transformer encoder-decoder architecture. It employs multi-codebook prediction (typically 8 codebooks) with optional local transformer refinement for high-quality audio generation, and leverages techniques like attention priors, classifier-free guidance (CFG), and Group Relative Policy Optimization (GRPO) for improved alignment. The generated codecs are then converted to speech waveform using NanoCodec.

This model is ready for commercial use. <br>

Key Features of the model

• Multilingual Support — Synthesizes natural speech in English, French, Spanish, German, French, Vietnamese, Italian, Mandarin, Hindi, Japanese.
• Expressive Voices — Multiple voice options with emotional tones and gender variations including 4 proprietary voices and 1 public voice
• Text Normalization — Built-in text normalization for handling numbers, abbreviations, and special characters for all languages except Vietnamese

Explore more from NVIDIA:

• For the enterprise offering, see the MagpieTTS NIM which includes additional native voices in the supported languages, emotional speech capabilities, and optimized batch and latency inference pipeline.
• What is Nemotron?<br>
• NVIDIA Developer Nemotron<br>
• Build a voice-agent code repo with the model.<br>

Deployment Geography:

Global

Use Case: <br>

For streaming voice agent use-cases, For Offline speech generation from text, in multiple languages.

Model Architecture:

Architecture Type: Transformer Encoder, Transformer Decoder, Local Transformer, and feedforward layers <br>

Figure 1: MagpieTTS Model Architecture

Network Architecture:

1. Causal Transformer Encoder with 6 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer. <br>
2. Causal Transformer Decoder with 12 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer. <br>

Number of model parameters 3.57*10^8 <br>

Input: <br>

Input Type(s): Text <br> Input Format: String <br> Input Parameters: One-Dimensional (1D) <br>

Output: <br>

Output Type(s): Audio <br> Output Format: .wav file <br> Output Parameters: One-Dimensional (1D) <br> Other Properties Related to Output: Audio output with dimensions (B x T), where B is batch size and T is time dimension. <br>

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>

How to Use this Model

NeMo Installation

To train, fine-tune or perform TTS with this model, you will need to install NVIDIA NeMo. We recommend you install it after you've installed latest PyTorch version and Python version ≥ 3.10.12.

pip install nemo_toolkit[tts]@main
pip install kaldialign

The model is available for use in the NeMo Framework, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Method 1: Single TTS Inference

In this method, the model can be used to infer on a single (text, language) pair. Text Normalization can also be applied if needed for En, Es, De, Fr, It, Zh languages. Load the Open Source MagpieTTS checkpoint from Huggingface and call the do_tts(transcript, language, apply_TN) method. This returns the generated audio and the length of the audio.

from nemo.collections.tts.models import MagpieTTSModel

speaker_map = {
    "John": 0,
    "Sofia": 1,
    "Aria": 2,
    "Jason": 3,
    "Leo": 4
}
transcript = "Hello world from NeMo Text to Speech."
language = "en"
speaker = "Sofia"
speaker_idx = speaker_map[speaker]

model = MagpieTTSModel.from_pretrained("nvidia/magpie_tts_multilingual_357m")
audio, audio_len = model.do_tts(transcript, language=language, apply_TN=False, speaker_index=speaker_idx)

# To apply custom phoneme customization in supported languages like English
# Surround the IPA string in a '|' character and add a space token between each IPA character.
ipa_transcript = "Hello world from | ˈ n ɛ m o ʊ | Text to Speech."
audio, audio_len = model.do_tts(ipa_transcript, language=language, apply_TN=False, speaker_index=speaker_idx)

Method 2: Batch Inference

This section explains how to run batch inference and evaluation on MagpieTTS models using the examples/tts/magpietts_inference.py script.

Key Points

The MagpieTTS inference script supports:

• Batch inference from .nemo files or .ckpt checkpoints
• Optional evaluation with metrics (CER, WER, Speaker Similarity, UTMOSv2)
• Multiple datasets in a single run

Dataset Configuration (examples/tts/evalset_config.json)

The script requires a JSON configuration file that defines the metadata for the datasets to process.

Format

{
    "dataset_name_1": {
        "manifest_path": "/absolute/path/to/manifest.json",
        "audio_dir": "/",
        "feature_dir": null
    },
    "dataset_name_2": {
        "manifest_path": "/path/to/another_manifest.json",
        "audio_dir": "/base/audio/path",
        "feature_dir": "/path/to/features"
    }
}

Fields

Example

{
    "libritts_test_clean": {
        "manifest_path": "/data/libritts/test_clean_manifest.json",
        "audio_dir": "/",
        "feature_dir": null,
        "whisper_language": "en"
    },
    "vctk": {
        "manifest_path": "/data/vctk/manifest.json",
        "audio_dir": "/data/vctk/wav48",
        "feature_dir": null
    }
}

---

Manifest Format

The manifest is a JSON-lines file where each line is a JSON object representing one utterance.

Minimum Required Fields

For models with fixed speaker context embeddings (no audio/text conditioning needed):

{"audio_filepath": "/path/to/audio.wav", "text": "The transcript text.", "duration": 3.5}

Run Inference and Evaluation

# Basic inference (no evaluation)
python examples/tts/magpietts_inference.py \
    --nemo_files "nvidia/magpie_tts_multilingual_357m" \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path "nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps" \
    --use_cfg \
    --cfg_scale 2.5

# Inference with evaluation
python examples/tts/magpietts_inference.py \
    --nemo_files "nvidia/magpie_tts_multilingual_357m" \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path "nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps" \
    --run_evaluation \
    --use_cfg \
    --cfg_scale 2.5

Check Outputs

After running, you'll find:

• Generated audio files in <out_dir>/<checkpoint_name>/
• Evaluation metrics in metrics.json
• Visualization plots (if evaluation enabled)

---

Evaluation Metrics

When --run_evaluation is enabled, the following metrics are computed:

Try via API — No Setup Required

Synthesize speech using the hosted NVIDIA NIM API on Magpie TTS Multilingual — no local GPU, Docker, or checkpoint download required.

1. Get a free API key: Open Magpie TTS Multilingual and choose Get API Key.

2. Install the Riva client:

pip install nvidia-riva-client

3. Synthesize speech to a WAV file:

import wave

import riva.client
from riva.client.proto.riva_audio_pb2 import AudioEncoding

auth = riva.client.Auth(
    uri="grpc.nvcf.nvidia.com:443",
    use_ssl=True,
    metadata_args=[
        ["function-id", "877104f7-e885-42b9-8de8-f6e4c6303969"],
        ["authorization", "Bearer nvapi-YOUR_API_KEY"],
    ],
)

service = riva.client.SpeechSynthesisService(auth)

sample_rate_hz = 22050
resp = service.synthesize(
    "Hello from the Magpie multilingual hosted API.",
    "Magpie-Multilingual.EN-US.Sofia",
    "en-US",
    sample_rate_hz=sample_rate_hz,
    encoding=AudioEncoding.LINEAR_PCM,
)

with wave.open("out.wav", "wb") as wf:
    wf.setnchannels(1)
    wf.setsampwidth(2)
    wf.setframerate(sample_rate_hz)
    wf.writeframesraw(resp.audio)

Or use the CLI (list voices, then synthesize):

git clone https://github.com/nvidia-riva/python-clients.git
export NVIDIA_API_KEY="nvapi-YOUR_API_KEY"

python python-clients/scripts/tts/talk.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "877104f7-e885-42b9-8de8-f6e4c6303969" \
    --metadata authorization "Bearer $NVIDIA_API_KEY" \
    --list-voices

python python-clients/scripts/tts/talk.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "877104f7-e885-42b9-8de8-f6e4c6303969" \
    --metadata authorization "Bearer $NVIDIA_API_KEY" \
    --text "Hello from Magpie." \
    --voice "Magpie-Multilingual.EN-US.Sofia" \
    --language-code en-US \
    --sample-rate-hz 22050 \
    -o out.wav

Note: Voice IDs and supported sample rates follow the deployed NIM. Use --list-voices against this endpoint, or see the API Reference on the Magpie TTS Multilingual page.

Software Integration:

Runtime Engine(s): NeMo Framework 25.11

Supported Hardware Microarchitecture Compatibility: <br>

• NVIDIA A10 GPU <br>
• NVIDIA A30 GPU <br>
• NVIDIA A100 GPU <br>
• NVIDIA H100 GPU <br>

Preferred/Supported Operating System(s):

• Linux <br>
• Linux 4 Tegra <br>

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>

Model Version(s):

Multilingual MagpieTTS-357M v2602<br>

Training and Evaluation Datasets:

Training Dataset:

The following datasets were used to train the model, including additional datasets focused on speech and ASR.

• Hi-FiTTS En<br>
• HiFiTTS-2 A Large-Scale High Bandwidth Speech Dataset En<br>
• LibriTTS En<br>
• Internal English Dataset
• CML-TTS Es
• Internal Spanish Dataset
• CML-TTS Fr
• Internal French Dataset
• CML-TTS It
• CML-TTS De
• Large-scale Vietnamese speech corpus (LSVSC) Vi
• InfoRe-2 Vi
• InfoRe-1 Vi
• Internal Vietnamese Dataset
• Internal Mandarin Dataset
• AI4bharat Hi
• Emilia YODAS Ja
• Common Voice Ja
• Internal Japanese Dataset

Data Modality <br>

• Audio <br>

<!-- 291.6 hifi, 36K hifi2, 585 Libri -->

Audio Training Data Size <br>

• 50,000 Hours <br>

Data Collection Method by dataset <br>

• Publicly available dataset <br>
• Human <br>

Labeling Method by dataset <br>

• Hybrid: Human, Synthetic - Human recorded data points were preprocessed algorithmically. <br>

Properties: Number of data items in training set: 38k hours Modality: Audio (speech signal) Nature of the content: Audio books Language: Multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja) Sensor Type: Microphones <br>

Evaluation Dataset:

• LibriTTS test-clean<br>
• CML-TTS Es<br>
• CML-TTS Fr<br>
• CML-TTS De<br>

Benchmark Score <br>

Data Collection Method by dataset: <br>

• Publicly available dataset <br>
• Human <br>

Labeling Method by dataset: <br>

• Human <br>
• Hybrid: Human, Synthetic - Human labeled data points are mixed and matched to create more variabilities. <br>

Properties: Modality: Audio (speech signal) Nature of the content: Audio books and Newspaper passages Language: Multilingual (En, Es, De, Fr) Sensor Type: Microphones <br>

• This result is based on the MagpieTTS model (Huggingface Checkpoint)

Inference:

Acceleration Engine: None <br> Test Hardware: <br>

• NVIDIA H100 GPU <br>
• NVIDIA A100 GPU <br>
• NVIDIA A6000 GPU <br>
• NVIDIA T4 GPU <br>

Technical Limitations & Mitigation:

There are two modes of inference, namely, standard and long-form. In standard mode, this model can generate up to twenty (20) seconds of multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja) speech at a time. In long-form mode, the model performs optimally when the input text contains punctuation and capitalization. The model was trained on a mix of publicly available speech datasets and internally recorded datasets in nine languages. As a result, it is not suitable for speech generation in any language other than the nine languages mentioned. We have removed zero-shot capabilities of this model for this release. Text normalization is required.

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/ . <br>

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by NVIDIA Open Model License Agreement

References(s):

1. Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment <br>
2. Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance <br>