wavlm-vocoder-french

wavlm-vocoder-french

News — June 2026: 🎉 The paper is now published in the JEP 2026 proceedings — 📄 read it here.

News — March 2026: This work was accepted at JEP 2026.

This repository hosts the main checkpoint associated with the paper, together with evaluation results, usage instructions, and links to the public demo and code.

A research checkpoint for French speech reconstruction from frozen WavLM representations, released as the companion model of our JEP 2026 accepted paper on WavLM-to-Audio vocoding for French speech.

🔗 Code: github.com/hi-paris/wavlm-vocoder-french
🎧 Demo: hi-paris.github.io/wavlm2audio-demo

---

Associated paper

WavLM-to-Audio Vocoding in French: Layer Ablation Study and Adversarial Supervision for Continuous Voice Conversion
Nassima Ould Ouali, Awais Hussein Sani, Reda Dehak, Eric Moulines
🔬 Model developed and trained by Nassima Ould Ouali — Research Engineer Hi! Paris / Ecole Polytechnique
Accepted at JEP 2026

This paper studies:

• layer-wise WavLM conditioning
• learned weighted fusion of WavLM layers
• adversarial versus non-adversarial training
• the feasibility of faithful French speech reconstruction from frozen self-supervised speech representations

---

🚀 Try the model

# 1. Clone the codebase
git clone https://github.com/hi-paris/wavlm-vocoder-french.git
cd wavlm-vocoder-french
pip install -e .

# 2. Run inference — checkpoint downloads automatically from HF Hub
python scripts/infer.py \
  --hf_repo hi-paris/wavlm-vocoder-french \
  --hf_filename checkpoint_step180000_infer.pt \
  --input_file /path/to/your/audio.wav \
  --output_dir ./generated \
  --device cuda

That's it. The checkpoint is downloaded automatically on first run.
Output: ./generated/{filename}_reconstructed.wav

---

Overview

This repository presents a WavLM-to-Audio neural vocoder for French speech. The system reconstructs waveform audio from frozen internal representations extracted from WavLM-Base+, using a learnable layer fusion mechanism, a convolutional adapter, and a HiFi-GAN-style neural generator.

This work is positioned as a foundational stage-1 component: before performing transformations in a continuous self-supervised latent space for voice conversion, it is necessary to ensure that the latent representation can first be decoded back into faithful audio.

---

Why this project matters

Modern voice conversion increasingly relies on continuous latent representations extracted from large self-supervised speech models such as Wav2Vec 2.0, HuBERT, and WavLM. However, reliable waveform reconstruction from such representations remains a technical challenge, especially for French speech.

This repository documents a French WavLM-to-Audio vocoder developed within a broader research effort on modular voice conversion. It provides a reconstructive decoder that can serve as a basis for future continuous voice conversion systems operating in the WavLM latent space.

---

Main idea

The architecture follows a three-stage pipeline:

1. Frozen WavLM-Base+ extracts internal speech representations
2. Learned weighted layer fusion + convolutional adapter combines and adapts the selected WavLM layers
3. A HiFi-GAN-style generator reconstructs the waveform at 16 kHz

The model explores:

• fixed last-layer conditioning
• averaging of the last N WavLM layers
• learned weighted fusion of all WavLM layers
• training with and without adversarial supervision (MPD/MSD + Feature Matching)

---

Selected demo checkpoint

The main checkpoint released in this repository is checkpoint_step180000_infer.pt (427 MB, inference-ready, generator weights only).

It corresponds to the best model from the JEP 2026 paper: full GAN supervision (MPD + MSD + Feature Matching) with learned weighted layer fusion.

The quantitative results reported below summarize the evaluation setting used in the accepted paper on a short French held-out reconstruction benchmark.

---

Training data

The model was trained on cleaned French speech data built from public corpora:

The data preparation pipeline includes mono conversion, resampling to 16 kHz, amplitude normalization, duration filtering, silence filtering, acoustic quality control, and manual inspection.

---

Model configuration

---

Evaluation summary

The model is evaluated in a short-audio reconstruction setting on a stratified subset of 15 test utterances covering unseen speakers, variable durations (1.5 s to 5 s), and diverse phonetic content.

These results correspond to the evaluation setting reported in the accepted JEP 2026 paper.

Impact of adversarial supervision

Adversarial supervision is critical for improving spectral quality, intelligibility, and prosodic fidelity in WavLM-to-Audio reconstruction.

---

Inference

Quick start (automatic download)

git clone https://github.com/hi-paris/wavlm-vocoder-french.git
cd wavlm-vocoder-french
pip install -e .

# Single file
python scripts/infer.py \
  --hf_repo hi-paris/wavlm-vocoder-french \
  --hf_filename checkpoint_step180000_infer.pt \
  --input_file /path/to/audio.wav \
  --output_dir ./generated \
  --device cuda

# Directory of files
python scripts/infer.py \
  --hf_repo hi-paris/wavlm-vocoder-french \
  --hf_filename checkpoint_step180000_infer.pt \
  --input_dir /path/to/audio_dir \
  --output_dir ./generated \
  --device cuda

Local checkpoint

python scripts/infer.py \
  --checkpoint /path/to/checkpoint_step180000_infer.pt \
  --input_file /path/to/audio.wav \
  --output_dir ./generated \
  --device cuda

Recommended input conditions

• Short French speech segments
• Mono audio at 16 kHz (or audio resampleable to 16 kHz)
• Short-duration utterances (1.5 s to 5 s) for best consistency with evaluation setting

Expected output

The inference script generates reconstructed waveforms saved as {stem}_reconstructed.wav in the specified output directory.

---

Intended use

• Research on French speech reconstruction from self-supervised latent representations
• Stage-1 decoding experiments for future continuous voice conversion
• Analysis of WavLM latent representations for speech generation
• Demonstration of a French WavLM-based vocoder

---

Limitations

This model is not a full voice conversion system. It is a reconstructive neural decoder designed as a foundational component.

• No explicit speaker identity conversion
• No explicit style transfer
• No prosody control module
• Evaluation primarily on short audio segments
• Stage-1 system only — not a complete end-to-end VC pipeline

---

Project context

This model corresponds to the reconstructive decoding stage required before future experiments on continuous latent-space transformation and controllable voice conversion. The associated paper studies layer-wise WavLM conditioning, learned weighted layer fusion, the effect of adversarial supervision, and the feasibility of faithful French speech reconstruction from frozen WavLM representations.

---

Citation

@misc{ouldouali2026wavlm2audiofr,
  title={WavLM-to-Audio Vocoding in French: Layer Ablation Study and Adversarial Supervision for Continuous Voice Conversion},
  author={Nassima Ould Ouali and Awais Hussain Sani and Eric Moulines and Reda Dehak},
  year={2026},
  note={Accepted at JEP 2026}
}