MERaLiON-OmniVoice-Hokkien-TTS

<h1 align="center">🔥 MERaLiON-TTS 🔥</h1> <p align="center"> <a href="https://huggingface.co/MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS">🚀 MERaLiON-OmniVoice-Hokkien-TTS</a> </p>

MERaLiON-OmniVoice-Hokkien-TTS is a fine-tuned text-to-speech model that generates natural-sounding Singapore Hokkien (Southern Min / 闽南语) speech from Chinese character input. It is built on OmniVoice and fine-tuned on Hokkien conversational data using SFT (Supervised Finetuning) and GRPO (Group Relative Policy Optimization).

Hokkien is an under-resourced language in the TTS space. Many Chinese TTS systems produce Mandarin pronunciation even when given Hokkien text written in Chinese characters. This model addresses that gap by generating speech with authentic Hokkien pronunciation, tones, and prosody.

Model Details

Demo Samples

Inference

Requirements

pip install torch transformers soundfile
pip install omnivoice  # or clone https://huggingface.co/k2-fsa/OmniVoice

Basic Usage

import torch
import soundfile as sf
from omnivoice.models.omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS",
    device_map="cuda:0",
    dtype=torch.float16,
)

text = "你阿母煮的菜真好食"
audios = model.generate(text=text, language="nan")
sf.write("output.wav", audios[0], model.sampling_rate)

Voice Cloning

For best results, provide a short reference audio clip (3-8 seconds) of the target speaker along with its transcript. The model will clone the speaker's voice characteristics.

import torch
import soundfile as sf
from omnivoice.models.omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS",
    device_map="cuda:0",
    dtype=torch.float16,
)

# Create a reusable voice clone prompt (process once, use many times)
voice_prompt = model.create_voice_clone_prompt(
    ref_audio="path/to/reference_speaker.wav",
    ref_text="Reference transcript in Hokkien",
)

# Generate with the cloned voice
text = "你阿母煮的菜真好食"
audios = model.generate(
    text=text,
    language="nan",
    voice_clone_prompt=voice_prompt,
)
sf.write("output.wav", audios[0], model.sampling_rate)

Tips:

• Use a clean, single-speaker reference clip at 16kHz mono WAV.
• The reference text should match what is spoken in the reference audio.
• language="nan" specifies Southern Min (ISO 639-3 code for Hokkien).
• create_voice_clone_prompt() encodes the reference once; reuse the returned prompt across multiple generations to avoid redundant processing.

Evaluation

We conducted a systematic evaluation comparing MERaLiON-OmniVoice-Hokkien-TTS against six other TTS systems on a benchmark of 25 Hokkien sentences. The evaluation covers four complementary dimensions: language correctness, transcription accuracy, perceived naturalness, and signal quality.

Test Set

The test set consists of 25 Singapore Hokkien sentences written in Chinese characters, ranging from short phrases (3-4 characters) to complex sentences (20+ characters). They cover everyday topics — greetings, directions, medical, food — and include Malay loanwords (e.g., "suka") that are characteristic of Singapore Hokkien.

We plan to expand evaluation with a lager phonetically-balanced test set and human MOS in future updates.

<details> <summary>Full test set (25 sentences)</summary> </details>

Compared Systems

All local models used the same Hokkien reference audio for voice cloning where supported.

Metrics

1. Language Identification (LangID)

What: An audio-based classifier determines whether each generated utterance sounds like Hokkien or Mandarin.

Why: The core challenge for Hokkien TTS is that most systems default to Mandarin pronunciation when reading Chinese characters. A model that produces Mandarin speech — regardless of how accurate or natural it sounds — has fundamentally failed at the task. LangID is therefore a gate metric: it tells us whether the model is even attempting the right language.

How: Qwen 3.5 Omni Plus listens to each audio clip and classifies it as Hokkien or Mandarin based on pronunciation and tonal patterns (not text content). The prompt explicitly instructs the model to focus on phonetic features and to treat Singapore Hokkien loanwords as Hokkien.

We chose Qwen 3.5 Omni Plus over Gemini 3.1 Pro because the former performed better on Hokkien ASR based on our tests. This also influenced our choice of Qwen 3.5 Omni Plus for WER and Naturalness.

2. Word Error Rate (WER)

What: Character-level WER between the original text and an ASR transcription of the generated audio.

Why: WER measures whether the model is saying the right words. A low WER means the generated speech faithfully reproduces the input text. We compute character-level WER (each Chinese character is a token) because Hokkien text has no standard word segmentation.

How: We use two independent ASR systems for robustness:

• Qwen 3.5 Omni Plus — prompted as a Hokkien transcription expert. Samples classified as Mandarin by LangID are penalized to WER=1.0 (since producing the wrong language is a total failure regardless of word accuracy).
• MERaLiON-3-3B-ASR — a local multilingual speech model. We report both raw scores and scores with the same LangID penalty applied.

3. Naturalness (MOS-like)

What: A 1–10 rating of how natural and human-like each utterance sounds.

Why: A model can pronounce words correctly but still sound robotic, choppy, or unnatural. Naturalness captures the perceptual quality — prosody, rhythm, fluency, voice quality, and absence of artifacts — that determines whether the speech is pleasant to listen to.

How: Qwen 3.5 Omni Plus rates each clip on a 1–10 scale, evaluating prosody, fluency, voice quality, pronunciation correctness, and artifacts. The evaluator is prompted with expertise in Hokkien and Chinese dialects.

4. DNSMOS P.835

What: An objective signal-quality score (1–5 scale) predicted by Microsoft's DNSMOS model.

Why: DNSMOS provides a language-agnostic signal quality assessment. It measures speech distortion (SIG), background noise (BAK), and overall quality (OVRL) without any dependence on language understanding. This complements the LLM-based metrics by catching low-level audio quality issues (clipping, noise, codec artifacts) that a language model might overlook.

How: Microsoft's pre-trained ONNX model (P.835 framework) runs locally on each audio file. No API calls or language-specific configuration needed.

Results

Summary Table

WER-Qwen and WER-MERaLiON both include the LangID penalty (Mandarin → WER=1.0). DNSMOS is the Overall (OVRL) score on the P.835 scale (1–5).

Key Findings

MERaLiON-OmniVoice-Hokkien-TTS achieves the best overall balance across all metrics. It is a local model with near-perfect language identification (24/25 Hokkien), the lowest WER (0.33/0.31), the highest naturalness (8.40), and competitive DNSMOS (3.13). It performs well across both language-aware and language-agnostic evaluation dimensions.

Most general-purpose TTS systems fail at Hokkien. Qwen3 TTS produces Mandarin 72% of the time (7/25 Hokkien). CosyVoice3 and CosyVoice Cloud also frequently default to Mandarin pronunciation. These models achieve high DNSMOS scores because the audio is clean — but they are speaking the wrong language.

Qwen3 TTS Flash is a strong cloud baseline. It achieves 25/25 Hokkien identification, 0.40 WER, 8.12 naturalness, and 3.07 DNSMOS. However, it requires API access and is not available for local deployment.

The Qwen naturalness evaluator is language-aware. Unlike Gemini Flash (which rated all models 8+), Qwen 3.5 Omni Plus assigns low naturalness scores (1.76–3.00) to models that produce Mandarin instead of Hokkien. This suggests it evaluates pronunciation correctness as part of naturalness, making it a more discriminating evaluator for dialect TTS.

Two independent ASR systems agree on rankings. The Qwen-based and MERaLiON-based WER scores produce consistent rankings when the LangID penalty is applied, lending confidence to the results despite the absence of a dedicated Hokkien ASR ground truth.

Detailed DNSMOS Breakdown

Evaluation Limitations

• No human evaluation. All subjective metrics (LangID, WER, naturalness) rely on LLM judges or ASR models, which may have systematic biases.
• Small test set. 25 sentences provide directional signal but are not sufficient for statistical significance claims.
• No dedicated Hokkien ASR. Both WER evaluators (Qwen, MERaLiON) were not specifically trained for Hokkien transcription, so absolute WER values should be interpreted as relative comparisons rather than ground truth error rates.
• Single-speaker evaluation. All voice-cloned models use the same reference speaker. Results may vary with different reference voices.

Citation

If you use this model or evaluation framework, please cite and attribute this work by linking back to this repository:

@misc{omnivoice-hokkien-2025,
  title={MERaLiON-OmniVoice-Hokkien-TTS: Fine-tuned Text-to-Speech for Singapore Hokkien},
  author={Lewis Won, MERaLiON Team},
  year={2026},
  url={https://huggingface.co/MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS}
}

Model Files

This repository contains the complete model — no additional downloads required:

├── config.json                      # Model architecture config
├── model.safetensors                # Fine-tuned LLM weights (1.9 GB)
├── tokenizer.json                   # Text tokenizer
├── tokenizer_config.json
├── chat_template.jinja
└── audio_tokenizer/                 # Bundled HiGGS Audio v2 tokenizer (769 MB)
    ├── config.json
    ├── model.safetensors
    └── preprocessor_config.json

Total size: ~2.7 GB.

License

Please refer to the MERaLiON-3 Public Licence for model terms.