arabic-tts-xtts-v2

Arabic TTS: Improving XTTS-v2 for Arabic Speech Synthesis

A systematic project to improve Coqui XTTS-v2 for high-quality Arabic text-to-speech generation. This repository documents the full journey from baseline evaluation to fine-tuned deployment, with reproducible benchmarks at every stage.

Table of Contents

• Project Overview
• Where This Project Sits in AI
• Architecture
• Setup & Installation
• Phase 1: Baseline Evaluation
• Phase 2: Compatibility Fixes
• Phase 3: Arabic Text Preprocessing
• Phase 4: Inference Pipeline Improvements
• Phase 5: Data Collection & Preparation
• Phase 6: Fine-Tuning
• Phase 7: Evaluation & Comparison
• Phase 8: Studio-Quality Audio Upsampling
• Resource Usage
• Results
• Known Limitations & Future Improvements
• Industry Use Cases
• Usage
• License

---

Project Overview

XTTS-v2 is a powerful multilingual TTS model supporting 17 languages out of the box — a remarkable achievement by the Coqui team. This project builds on that foundation to push Arabic output quality further by:

1. Expanding Arabic training data — supplementing the base model with additional high-quality Arabic speech data
2. Adding Arabic text preprocessing — introducing diacritization (tashkeel) and hamza normalization to help the model with Arabic-specific orthography
3. Optimizing the tokenizer for Arabic — adapting the shared multilingual tokenizer to better handle Arabic text patterns
4. Tuning inference for Arabic prosody — adjusting generation parameters to better match Arabic speech rhythms and intonation
5. Building a post-processing pipeline — adding pause control, silence management, and output refinement tailored to Arabic

This project addresses each area systematically, with before/after comparisons at every stage.

Goals

• Natural-sounding Arabic speech — produce voices that sound authentically human, not robotic or accented
• Multi-dialect support — generate speech across different Arabic dialects and accents (e.g. Sudanese, Egyptian, Gulf, Levantine, North African, MSA) by fine-tuning on dialect-specific speakers
• Document a reproducible pipeline from base model to production-quality Arabic TTS
• Provide quantitative benchmarks (MOS scores, RTF, SNR) at each improvement stage
• Release fine-tuned checkpoints and preprocessing tools for the Arabic TTS community

---

Where This Project Sits in AI

Figure 1: The relationship between AI, Machine Learning, Deep Learning, and Data Science — and where each part of this project applies.

This project spans all four domains. Here is where each phase of our work falls:

In detail:

• Deep Learning powers the core model — the GPT-2 backbone generates audio codes autoregressively, the HiFiGAN vocoder converts them to waveforms, and the Perceiver Resampler handles speaker conditioning. Fine-tuning these neural networks on Arabic data is a Deep Learning task
• Machine Learning solved our speaker identification problem — with 103K unlabeled clips, we used ECAPA-TDNN (a neural feature extractor) combined with Agglomerative Clustering (a classical ML algorithm) to find and extract 5,000 clips from the same speaker
• Data Science drove every decision about data quality — from evaluating 6 candidate datasets, to measuring SNR and silence ratios, to building the 3-layer sanity check pipeline, to generating comparison charts and metrics
• AI is the umbrella — the final system takes Arabic text as input and produces human-like speech as output, combining all three disciplines into a practical application

---

Architecture

XTTS-v2 Model Components

Figure 2: XTTS-v2 architecture — text and reference audio are processed through separate pathways, combined in the GPT-2 transformer, and decoded to a 24kHz waveform.

Key Specifications

---

Setup & Installation

Hardware

This project was developed on an NVIDIA DGX Spark — a desktop AI workstation with a Grace Blackwell architecture. If you're using different hardware, you may need to adjust CUDA versions, PyTorch builds, and batch sizes accordingly.

[!NOTE]

Running on different hardware?

Consumer GPUs (RTX 3090, 4090, etc.):

• You'll need PyTorch built for your CUDA version (e.g., cu118, cu121, cu124 instead of cu130)
• With 24GB VRAM, reduce batch_size to 2 and grad_accumulation_steps to 1 during fine-tuning
• Inference works fine on 8GB+ VRAM

Data Center GPUs (A100, H100):

• Should work out of the box with matching CUDA/PyTorch versions
• Can increase batch sizes for faster training

x86 vs ARM (aarch64):

• DGX Spark uses ARM. If you're on x86 (most desktops/servers), install the x86 PyTorch build — the code is the same, only the binary packages differ
• pip install TTS will automatically pull the correct architecture

CPU only:

• Inference is possible but very slow (~10x slower than GPU)
• Fine-tuning on CPU is not practical

Key: Match your PyTorch version to your CUDA version. Check compatibility at pytorch.org/get-started.

Software Environment

Installation Steps

# 1. Create conda environment
conda create -n new-arabic-tts python=3.10 -y
conda activate new-arabic-tts

# 2. Install Coqui TTS
pip install TTS

# 3. Pin transformers for compatibility (see Phase 2)
pip install transformers==4.44.2

# 4. Download XTTS-v2 base model
echo "y" | python -c "
from TTS.utils.manage import ModelManager
mm = ModelManager()
mm.download_model('tts_models/multilingual/multi-dataset/xtts_v2')
"

# 5. Copy model to project directory
mkdir -p models/base
cp -r ~/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2/* models/base/

Project Structure

New Arabic TTS/
├── README.md                       # This file
├── CHANGELOG.md                    # Detailed change log
├── configs/                        # Configuration files
├── data/
│   └── egyptian/                   # Phase 5: Prepared training data
│       ├── metadata_train.csv      #   Training split (cleaned)
│       ├── metadata_eval.csv       #   Evaluation split (cleaned)
│       ├── wavs/                   #   22,050 Hz mono WAV files
│       └── rejected/              #   Clips that failed sanity check
├── docs/
│   ├── DATA_FORMAT.md              # Data format guide for fine-tuning
│   ├── images/
│   │   ├── speaker_clusters_umap.png   # Phase 5: UMAP speaker clusters
│   │   ├── cluster_sizes.png           # Phase 5: Cluster size chart
│   │   ├── similarity_distribution.png # Phase 5: Speaker consistency cutoff
│   │   ├── duration_distribution.png   # Phase 5: Duration histogram
│   │   ├── comparison_waveforms.png    # Phase 7: Waveform comparison
│   │   ├── comparison_spectrogram.png  # Phase 7: Spectrogram comparison
│   │   ├── comparison_metrics.png      # Phase 7: Metrics bar chart
│   │   └── training_loss.png           # Phase 7: Training loss curve
│   └── benchmarks/
│       ├── baseline.json               # Phase 1: Baseline metrics
│       ├── dataset_preparation.json    # Phase 5: Pipeline report
│       ├── sanity_check.json           # Phase 5: Data quality report
│       ├── training_summary.json       # Phase 6: Training results
│       └── evaluation.json             # Phase 7: Comparison metrics
├── models/
│   └── base/                       # XTTS-v2 base model (1.8GB)
│       ├── config.json
│       ├── model.pth
│       ├── speakers_xtts.pth       # 58 built-in speakers
│       └── vocab.json
├── outputs/
│   ├── original model/             # Phase 1: Raw base model outputs
│   │   └── Original_Model_test.wav #   Baseline benchmark (standard demo text)
│   ├── improved model/             # Phase 3+4: Text preprocessing + post-processing
│   │   ├── Improved_Model_test.wav #   Same text, improved pipeline
│   │   └── Improved_Model_test_stats.json  # Per-chunk generation stats
│   └── finetuned model/             # Phase 6: After fine-tuning
│       ├── Finetuned_Model_test.wav          # Fine-tuned output (same demo text)
│       └── Finetuned_Model_test_stats.json   # Generation stats
└── scripts/
    ├── arabic_preprocessor.py      # Phase 3: Arabic text preprocessing
    ├── baseline_test.py            # Phase 1: Baseline benchmark generator
    ├── infer.py                    # Phase 4: Improved inference pipeline
    ├── patch_tts.py                # Phase 2: PyTorch compatibility patch
    ├── prepare_dataset.py          # Phase 5: Dataset download, clustering, export
    ├── refine_cluster.py           # Phase 5: Refined top-N speaker selection
    ├── sanity_check.py             # Phase 5: 3-layer data quality verification
    ├── train.py                    # Phase 6: XTTS-v2 fine-tuning
    ├── evaluate.py                 # Phase 7: Metrics and comparison charts
    └── upsample.py                 # Phase 8: AudioSR upsampling (24→48kHz)

---

Phase 1: Baseline Evaluation

Objective: Establish baseline Arabic output quality using the unmodified XTTS-v2 model.

1.1 Model Download

Downloaded XTTS-v2 base model (1.8GB) from Coqui's model hub. The model includes:

• model.pth - Full checkpoint (~370M parameters)
• speakers_xtts.pth - 58 pre-computed speaker embeddings
• vocab.json - BPE vocabulary (6,681 tokens)
• config.json - Model and training configuration

1.2 Built-in Speakers

XTTS-v2 ships with 58 speaker embeddings. No Arabic-native speakers are included - all speakers are from English/European datasets. We selected "Gilberto Mathias" for baseline testing as a speaker with a name suggesting potential Arabic phonetic compatibility.

<details> <summary>Full speaker list (58 speakers)</summary>

Aaron Dreschner, Abrahan Mack, Adde Michal, Alexandra Hisakawa, Alison Dietlinde, Alma Maria, Ana Florence, Andrew Chipper, Annmarie Nele, Asya Anara, Gilberto Mathias, Baldur Sanjin, Barbora MacLean, Brenda Stern, Camilla Holmstrom, Chandra MacFarland, Claribel Dervla, Craig Gutsy, Daisy Studious, Damien Black, Damjan Chapman, Dionisio Schuyler, Eugenio Mataraci, Ferran Simen, Filip Traverse, Gilberto Mathias, Gitta Nikolina, Gracie Wise, Henriette Usha, Ige Behringer, Ilkin Urbano, Kazuhiko Atallah, Kumar Dahl, Lidiya Szekeres, Lilya Stainthorpe, Ludvig Milivoj, Luis Moray, Maja Ruoho, Marcos Rudaski, Narelle Moon, Nova Hogarth, Rosemary Okafor, Royston Min, Sofia Hellen, Suad Qasim, Szofi Granger, Tammie Ema, Tammy Grit, Tanja Adelina, Torcull Diarmuid, Uta Obando, Viktor Eka, Viktor Menelaos, Vjollca Johnnie, Wulf Carlevaro, Xavier Hayasaka, Zacharie Aimilios, Zofija Kendrick

</details>

1.3 Baseline Test

Standard demo text — this text is used consistently across all phases for comparison. It is also the default demo text when the model is published on HuggingFace:

مرحباً بكم في عالم التكنولوجيا الحديثة.
تُعدّ اللغة العربية من أقدم اللغات وأكثرها انتشاراً في العالم،
إذ يتحدث بها أكثر من أربعمائة مليون شخص حول العالم.

Generation parameters (baseline):

Output file: outputs/original model/Original_Model_test.wav — 19.33 seconds Benchmark data: docs/benchmarks/baseline.json

Listen to this file to hear the unmodified XTTS-v2 base model generating Arabic. This is the starting point — all subsequent improvements are measured against this output.

1.4 Baseline Observations

Note: Detailed quality metrics (MOS, SNR, intelligibility) will be added after the evaluation pipeline is built in Phase 7.

Known issues with baseline output:

• Pronunciation errors on undiacritized words
• Non-native Arabic accent (English-trained speaker embedding)
• Inconsistent prosody and pacing
• No pause control between sentences (fixed 0.35s gaps)
• Potential rambling/over-generation on longer sentences

---

Phase 2: Compatibility Fixes

Before any improvements could begin, two compatibility issues had to be resolved.

2.1 Transformers Version Conflict

Problem: Coqui TTS 0.22.0 imports BeamSearchScorer from HuggingFace Transformers, which was removed in Transformers v5.x.

ImportError: cannot import name 'BeamSearchScorer' from 'transformers'

Fix: Pin Transformers to a compatible version.

pip install transformers==4.44.2

2.2 PyTorch weights_only Default Change

Problem: PyTorch 2.11 changed the default of torch.load() from weights_only=False to weights_only=True. XTTS-v2 checkpoints contain non-tensor objects (config classes) that fail to load under the new default.

_pickle.UnpicklingError: Weights only load failed.
Unsupported global: GLOBAL TTS.tts.configs.xtts_config.XttsConfig

Fix: Patched TTS/utils/io.py to explicitly pass weights_only=False.

# File: <env>/lib/python3.10/site-packages/TTS/utils/io.py
# Lines 51, 54: Added weights_only=False to both torch.load() calls

# Before:
return torch.load(f, map_location=map_location, **kwargs)

# After:
return torch.load(f, map_location=map_location, weights_only=False, **kwargs)

Note: This patch is applied to the installed package. If the new-arabic-tts conda environment is recreated, this patch must be re-applied. A script to automate this is provided at scripts/patch_tts.py.

2.3 Coqui Trainer Git Branch Detection

Problem: The Coqui Trainer class tries to detect the current git branch for logging. If the project directory is a git repo with no commits, git branch returns no output and the trainer crashes with StopIteration.

StopIteration (in trainer/generic_utils.py get_git_branch)

Fix: Initialize the git repo with at least one commit before running training.

git add . && git commit -m "Initial commit"

2.4 AudioSR Dependency Conflicts

Problem: Installing audiosr downgrades librosa (0.10→0.9.2) and transformers (4.44→4.30), breaking Coqui TTS compatibility.

Fix: After installing AudioSR, restore the required versions:

pip install audiosr
pip install "librosa>=0.10.0" "transformers==4.44.2"  # restore after audiosr

---

Phase 3: Arabic Text Preprocessing

Objective: Build an Arabic-aware text preprocessing pipeline that improves pronunciation without forcing a specific dialect.

Key design decision: Tashkeel (diacritization) is off by default. Adding diacritics forces formal MSA pronunciation, which conflicts with the project's goal of natural dialect-specific speech. Instead, the model learns pronunciation from the speaker's training data. Tashkeel is available as an optional flag (--tashkeel) for users who want formal Arabic output.

Script: scripts/arabic_preprocessor.py

3.1 Hamza Normalization

Corrects common hamza placement errors using a 40+ word correction map. This helps the tokenizer produce consistent token sequences regardless of how the input was typed.

Examples:

3.2 Number and Symbol Expansion

Converts numerals and symbols to Arabic words so the model doesn't have to guess pronunciation of non-text characters.

Examples:

3.3 Tashkeel (Optional — MSA Mode)

When enabled via --tashkeel, the Mishkal engine adds full diacritical marks. This produces formal MSA pronunciation.

Example:

Input:    الذكاء الاصطناعي يتطور بسرعة كبيرة
Tashkeel: الذَّكَاءُ الْاِصْطِنَاعِيُّ يَتَطَوَّرُ بِسُرْعَةِ كَبِيرَةِ

3.4 Processing Pipeline

Default (dialect mode):     clean → hamza → numbers → symbols → inference
Optional (formal MSA mode): clean → hamza → numbers → symbols → tashkeel → inference

---

Phase 4: Inference Pipeline Improvements

Objective: Build a production-quality inference pipeline with smart chunking, post-processing, and reproducible output.

Script: scripts/infer.py

4.1 Smart Text Chunking

The base XTTS-v2 has a 166-character limit per inference call. Our chunker handles this with a 3-tier strategy:

1. Sentence splitting — split at Arabic punctuation (. ؟ ! ؛)
2. Long sentence splitting — split at commas/semicolons if over 160 chars, then by word boundary as last resort
3. Short chunk merging — merge chunks under 30 chars with their neighbors to avoid choppy output

4.2 Post-Processing Pipeline

Each generated chunk goes through:

4.3 Deterministic Generation

Each chunk gets a unique seed (base_seed + chunk_index), so the same input always produces the same output. Default base seed: 12345.

4.4 Configurable Pause Control

4.5 First Improved Output

Generated the same AI test text through the improved pipeline:

Output file: outputs/improved model/Improved_Model_test.wav — 18.99 seconds Stats file: outputs/improved model/Improved_Model_test_stats.json

Listen to this file and compare with outputs/original model/Original_Model_test.wav to hear the effect of text preprocessing and post-processing. The voice is the same — the improvements are in how text is handled and how audio is cleaned.

[!IMPORTANT]

What improved and what didn't (yet)

Improved in this phase (pipeline & text):

• Numbers are spoken correctly (70% → سبعون بالمئة instead of raw symbols)
• Hamza correction ensures consistent tokenization (اكثر → أكثر)
• Cleaner audio: internal pauses capped at 150ms, natural sentence endings with taper
• Reproducible output via deterministic seeding
• Per-sentence stats tracked in JSON

Not improved yet (requires fine-tuning in Phase 6):

• Voice is still a non-Arabic built-in speaker (Gilberto Mathias)
• Pronunciation still reflects the base model's limited Arabic training
• Accent and prosody are not natural Arabic

The voice quality transformation happens in Phase 6 when we fine-tune on real Arabic speaker data. Phases 3–4 ensure the pipeline is clean and correct so that fine-tuning produces the best possible results.

4.6 Output Reference Guide

All output files are organized by phase. Each file captures a specific stage of improvement so you can listen and compare the progression:

How to compare: Play Original_Model_test.wav and Improved_Model_test.wav side by side — all files use the same standard demo text. After Phase 6, play the fine-tuned version to hear the voice quality transformation. This progression tells the full story of the project.

---

Phase 5: Data Collection & Preparation

Status: In Progress

For full details on the data format, see docs/DATA_FORMAT.md.

5.1 Data Format Summary

XTTS-v2 fine-tuning requires:

• Audio: WAV files, 22,050 Hz mono (auto-resampled), 0.5–11.6s per clip
• Metadata: Pipe-delimited CSV with header: audio_file|text|speaker_name
• Split: Separate metadata_train.csv and metadata_eval.csv

5.2 Dataset Evaluation

We evaluated all publicly available Arabic speech datasets for suitability as XTTS-v2 fine-tuning data. The ideal dataset has: single-speaker or speaker-labeled clips, 2–11s duration, clean audio, accurate transcripts, and 22kHz+ sample rate.

Fully Free & Ready to Download

Free with Registration

Other Arabic Speech Resources

Evaluation Summary

Figure 4: Dataset evaluation results — green (chosen), yellow (viable alternatives), red (rejected).

5.3 Chosen Approach

Primary dataset: MAdel121/arabic-egy-cleaned — starting with a subset for initial fine-tuning experiments.

Why: It's the largest freely available Egyptian Arabic dataset, already cleaned, with normalized transcripts that match our dialect-first approach (no diacritics). The 16kHz sample rate is a compromise, but XTTS-v2 handles resampling automatically.

Future: As the project expands to more dialects (Sudanese, Gulf, Levantine), we will collect dialect-specific data from single speakers — either from YouTube (using yt-dlp + Whisper for transcription) or from dedicated recording sessions.

5.4 Data Preparation Pipeline

The source dataset has 103K clips with no speaker labels. To fine-tune XTTS-v2 for a single consistent voice, we need clips from the same speaker. We built an automated speaker clustering pipeline to solve this.

Curated with care: The heavy lifting of speaker identification, quality filtering, and format conversion has been done for you. If you're using this project as a starting point, the exported data in data/egyptian/ is ready for training — no additional preparation needed.

Script: scripts/prepare_dataset.py Source dataset: MAdel121/arabic-egy-cleaned (HuggingFace)

Pipeline Overview

Figure 3: Data preparation pipeline — from raw HuggingFace dataset to XTTS-v2 training-ready format.

Step-by-Step Details

Step 1 — Download

Step 2 — Quality Filter

Removes clips unsuitable for XTTS-v2 fine-tuning before any expensive processing:

Step 3 — Speaker Embedding Extraction

Extracts a numerical "voice fingerprint" for each clip so we can identify who is speaking.

Why ECAPA-TDNN over alternatives?

Step 4 — Speaker Clustering

Groups embeddings so that each cluster represents clips from the same speaker.

Step 5 — Best Cluster Selection

Picks the cluster most likely to be a single consistent speaker.

Step 6 — Export to XTTS-v2 Format

Converts the selected cluster into the exact format XTTS-v2 expects for fine-tuning.

Generated Visualizations

The pipeline automatically generates analysis charts saved to docs/images/:

5.5 Reproducibility

To rerun the full pipeline from scratch:

conda activate new-arabic-tts
cd "New Arabic TTS"
python scripts/prepare_dataset.py

The dataset will be cached after the first download. Subsequent runs skip the download and go straight to filtering. All random operations use seed=42 for reproducibility.

5.6 Dependencies Added in This Phase

pip install datasets speechbrain umap-learn matplotlib torchcodec openai-whisper

[!NOTE] Why OpenAI Whisper instead of faster-whisper? We initially tried faster-whisper (CTranslate2-based) but it lacks CUDA support on ARM (aarch64) platforms like DGX Spark. OpenAI's original Whisper uses PyTorch directly and works with any CUDA-capable GPU.

5.7 Data Sanity Check

[!IMPORTANT]

Why clean data matters for TTS

TTS fine-tuning is fundamentally different from ASR (speech recognition). An ASR model can tolerate noisy data because it's learning to understand speech — a few bad transcripts won't ruin comprehension. But a TTS model learns to reproduce exactly what it hears:

• Misaligned text (text doesn't match audio) → the model learns wrong pronunciations and associates incorrect words with sounds
• Noisy audio → the model learns to generate background noise, hiss, or hum as part of normal speech
• Silence or dead air → the model learns to insert random pauses mid-sentence
• Corrupted or garbled text → the model learns to mispronounce words consistently

A few bad clips in a 5,000-clip dataset might go unnoticed. But hundreds of bad clips will measurably degrade every output the fine-tuned model produces. The quality of your training data directly determines the quality ceiling of your model.

Script: scripts/sanity_check.py

We apply a three-layer automated verification pipeline to catch problems before they reach the training stage.

Layer 1 — Text Quality

Fast checks on the transcript text to catch obviously bad entries.

Layer 2 — Audio Quality

Signal analysis on the audio waveform.

Layer 3 — Alignment Verification (Whisper)

The most important layer. Uses OpenAI Whisper (large-v3) to independently transcribe each clip, then compares the Whisper transcript to the metadata text.

Text normalization before comparison:

• Remove diacritics
• Normalize alef variants (أ/إ/آ → ا)
• Normalize taa marbuta (ة → ه)
• Remove punctuation

Output Files

How to run

conda activate new-arabic-tts
python scripts/sanity_check.py

Dependencies

pip install openai-whisper

Note: We use openai-whisper (PyTorch-based) instead of faster-whisper (CTranslate2-based) because CTranslate2 lacks CUDA support on ARM platforms (DGX Spark). If you're on x86 with CUDA, faster-whisper will also work and is faster.

---

Phase 6: Fine-Tuning

Status: In Progress

Script: scripts/train.py

6.1 Training Configuration

6.2 Training Data

6.3 Training Process

conda activate new-arabic-tts
python scripts/train.py

Output saved to models/finetuned/run/training/ with:

• Best model checkpoint
• TensorBoard logs
• Training configuration

6.4 Training Results

6.5 Fine-Tuned Output

Output file: outputs/finetuned model/Finetuned_Model_test.wav — 16.76 seconds Stats file: outputs/finetuned model/Finetuned_Model_test_stats.json

Listen to this file and compare with the baseline and improved outputs. The voice now comes from a real Arabic speaker (Egyptian male) rather than a built-in English-trained speaker. This is the core quality transformation of the project.

6.6 Full Comparison (Same Demo Text)

All three outputs use the same standard demo text for a fair comparison:

[!NOTE] The fine-tuned output is shorter (16.76s vs 19.33s) because the Arabic-trained speaker speaks at a more natural pace compared to the English-trained baseline speaker.

---

Phase 7: Evaluation & Comparison

Status: Complete

Script: scripts/evaluate.py

7.1 Evaluation Metrics

All three outputs were analyzed using the same standard demo text for a fair comparison.

Note on SNR: The baseline has higher SNR because the built-in speaker produces cleaner but less natural audio. The fine-tuned model has lower SNR because it reproduces the natural characteristics of real speech (breathing, vocal texture), which register as "noise" in a simple SNR measurement. This is expected and desirable — real human speech is not perfectly clean.

7.2 Generated Charts

All charts are saved to docs/images/ and generated automatically by scripts/evaluate.py.

Waveform Comparison

Side-by-side waveforms of the same text across all three stages. The fine-tuned model shows more natural amplitude variation and pacing.

Spectrogram Comparison

Frequency content across time. The fine-tuned model shows richer harmonic structure typical of natural Arabic speech.

Metrics Comparison

Bar chart comparing duration, SNR, speaking duration, and silence ratio across all stages.

Training Loss Curve

Loss curve during fine-tuning. Steady decrease from 4.47 to 3.60 over 4 epochs (4,328 steps, 30 minutes).

7.3 Audio Samples

Listen to all three outputs in order to hear the progression:

7.4 How to Reproduce

conda activate new-arabic-tts
python scripts/evaluate.py

This regenerates all charts and metrics from the existing output WAV files.

---

Phase 8: Studio-Quality Audio Upsampling

Status: Complete

[!IMPORTANT]

Why not change the model's internal sample rate?

XTTS-v2's sample rate (22,050 Hz internal → 24,000 Hz output) is deeply embedded across all components — the DVAE codec, GPT backbone, HiFiGAN vocoder, and mel spectrogram extraction are all trained and calibrated for these rates. Changing the output sample rate would require retraining the entire model from scratch (~350M parameters), which is not feasible.

Instead, we use a neural audio upsampler as a post-processing step after the model generates speech. This is the same approach used in professional TTS production pipelines.

8.1 Approach

XTTS-v2 (24 kHz / 16-bit) → Neural Upsampler → 48 kHz or 96 kHz / 24-bit

The upsampler reconstructs the high-frequency detail (4–24 kHz) that the base model cannot produce, resulting in studio-quality output without modifying the core model.

Audio Quality Tiers

8.2 Chosen Upsampler: AudioSR

We evaluated three options and chose AudioSR:

Script: scripts/upsample.py

8.3 Upsampling Process

All three official outputs are upsampled for comparison:

conda activate new-arabic-tts
python scripts/upsample.py --all

8.4 AudioSR Parameters

[!NOTE] AudioSR reconstructs the high-frequency content (8–24 kHz) that the base XTTS-v2 model cannot produce at its native 24kHz output. The result is noticeably crisper consonants, more natural breathiness, and professional-grade audio suitable for podcasts, voiceover, and video production.

8.5 Upsampling Results

The best output from this project is Finetuned_Model_test_48kHz.wav — fine-tuned Arabic voice at professional 48kHz/24-bit quality.

---

Resource Usage

Every phase was tracked for compute resources. This helps you estimate hardware requirements if running on different GPUs.

Per-Phase Resource Breakdown

GPU Memory Summary

┌──────────────────────────────────────────────────────────────┐
│  DGX Spark GPU: 121.7 GB unified memory                      │
│                                                              │
│  Peak usage by task:                                         │
│  ██░░░░░░░░░░░░░░░░░░  Training (22 GB)      18% of total   │
│  █░░░░░░░░░░░░░░░░░░░  Whisper check (10 GB)  8% of total   │
│  █░░░░░░░░░░░░░░░░░░░  AudioSR (6 GB)         5% of total   │
│  ░░░░░░░░░░░░░░░░░░░░  Inference (5 GB)       4% of total   │
│                                                              │
│  Minimum GPU for this project: 8 GB (inference only)         │
│  Minimum GPU for full pipeline: 24 GB (training + Whisper)   │
└──────────────────────────────────────────────────────────────┘

For Different Hardware

Total Pipeline Cost

---

Results

---

Usage

CLI Usage

# Basic generation (dialect mode — no tashkeel)
python scripts/infer.py --text "مرحباً بكم في العالم العربي" --output outputs/test.wav

# From a text file
python scripts/infer.py --text-file input.txt --output outputs/test.wav

# Formal MSA mode (with tashkeel)
python scripts/infer.py --text "مرحباً بكم" --output outputs/msa.wav --tashkeel

# Custom parameters
python scripts/infer.py --text "مرحباً" --output outputs/test.wav \
    --speaker "Gilberto Mathias" --temperature 0.55 --top-p 0.85 --rep-penalty 2.5 --pause 0.4

Python API

from scripts.infer import load_model, load_speaker, generate
from scripts.arabic_preprocessor import ArabicPreprocessor
import soundfile as sf

model = load_model("models/base")
gpt_cond_latent, speaker_embedding = load_speaker("models/base", "Gilberto Mathias")

# Dialect mode (default)
preprocessor = ArabicPreprocessor(enable_tashkeel=False)

result = generate(
    model=model,
    text="الذكاء الاصطناعي يتطور بسرعة كبيرة",
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
    preprocessor=preprocessor,
)

sf.write("output.wav", result["wav"], result["sr"])
print(result["stats"])

---

Known Limitations & Future Improvements

Current Limitations

The current model produces noticeably improved Arabic speech compared to the base XTTS-v2, but it is not yet at the level of a professional human voice actor. The primary bottleneck is data quality, not model architecture.

Why data is the limiting factor:

The training data used in this release comes from MAdel121/arabic-egy-cleaned, an open-source Egyptian Arabic speech dataset originally built for ASR (speech recognition), not TTS. While we applied extensive cleaning — speaker clustering, quality filtering, and Whisper-based alignment verification — the data has inherent limitations:

• No speaker labels in the source — our speaker clustering identifies the most consistent voice, but it's not guaranteed to be a single speaker across all 4,328 training clips
• 16 kHz source audio — upsampled to 22,050 Hz for training, but the high-frequency detail above 8 kHz was never captured in the original recordings
• ASR-grade transcripts — the text was generated by speech recognition models, not hand-verified. Our Whisper cross-check caught 188 misaligned clips (3.8%), but subtle errors may remain
• Single dialect — currently trained on Egyptian Arabic only. Sudanese, Gulf, Levantine, and other dialects require their own training data

Data Collection Pipeline (Next Phase)

To achieve production-quality Arabic TTS, we need purpose-built training data. The next phase will use a custom data collection pipeline:

Figure 5: Future data collection pipeline — from YouTube to training-ready data. Step 6 (Manual Review) is the most critical step for TTS data quality.

[!IMPORTANT] Manual cleaning is crucial. No automated pipeline — no matter how sophisticated — can replace a human listener verifying that text matches audio. The difference between a good TTS model and a great one often comes down to the last 5% of data quality that only manual review can catch.

Acknowledgments — Training Data

We gratefully acknowledge the following open-source data providers whose work made this project possible:

• MAdel121 — Creator of the arabic-egy-cleaned dataset (72h Egyptian Arabic, cleaned and normalized). This dataset served as our primary training data source
• Mozilla Common Voice — Crowdsourced multilingual speech data including Arabic contributions
• Coqui AI — Creators of XTTS-v2 and the Coqui TTS framework that this project builds upon
• SpeechBrain — ECAPA-TDNN speaker verification model used for speaker clustering
• OpenAI Whisper — Speech recognition model used for data alignment verification

Roadmap

---

Industry Use Cases

When this project reaches production quality with clean, manually verified training data across multiple Arabic dialects, it enables a wide range of applications:

Media & Entertainment

Business & Enterprise

Technology & AI

Education & Government

Market Opportunity

The Arabic-speaking world represents over 400 million native speakers across 25+ countries, yet Arabic TTS technology significantly lags behind English and European languages. High-quality, dialect-aware Arabic TTS addresses an underserved market with growing demand for Arabic digital content, accessibility tools, and AI-powered services.

[!NOTE]

Future Releases

This is an active project. Improved versions of the model will be released as we collect cleaner training data, expand dialect coverage, and refine the pipeline. Future releases will include:

• v2.0 — Retrained on manually verified, studio-quality Arabic speech data
• Dialect-specific checkpoints — Separate models for Sudanese, Egyptian, Gulf, Levantine, and MSA
• Pre-built speaker profiles — Ready-to-use voice embeddings for common use cases

Follow this repository for updates. Releases will be published on both GitHub and HuggingFace.

---

License

• XTTS-v2 Base Model: Coqui Public Model License (CPML)
• This Project: MIT License

---

Citation

@misc{arabic-tts-xtts-v2,
  title={Improving XTTS-v2 for Arabic Text-to-Speech},
  author={Moe Eldouma},
  year={2026},
  url={https://github.com/eldoumamoe-glitch/arabic-tts}
}