anogkongda_librimix_enh_train_raw_valid.si_snr.ave

Usage

pip install espnet
espnet enhance noisy.wav -o enhanced.wav --model espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave

import soundfile as sf
from espnet2.bin.enh_inference import SeparateSpeech

enh = SeparateSpeech.from_pretrained(model_tag="espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave")
# channels are kept on the way in: a multi-channel model needs them
speech, rate = sf.read("noisy.wav", dtype="float32")
waves = enh(speech[None, ...], fs=rate)
# one wave per output stream, which is what tells the two tasks apart: an
# enhancement model returns one, a separation model one per speaker. This is
# what `espnet enhance` writes as enhanced.wav or enhanced.spk1.wav, ...
if len(waves) == 1:
    sf.write("enhanced.wav", waves[0][0], rate)
else:
    for i, wave in enumerate(waves, start=1):
        sf.write(f"enhanced.spk{i}.wav", wave[0], rate)

Example audio

samples/ holds one Libri2Mix test mixture, so the snippet above can be run without building the corpus:

They are the pair the official Libri2Mix metadata names as mixture 7729-102255-0031_2094-142345-0028, in metadata/Libri2Mix/libri2mix_test-clean.csv. To rebuild that mixture, do what create_librimix_from_metadata.py does in min mode — apply the row's gains, resample to 8 kHz, truncate to the shorter source, and sum:

import numpy as np, soundfile as sf
from scipy.signal import resample_poly

GAIN = {"7729-102255-0031": 0.5159350719675272, "2094-142345-0028": 0.22360240150798563}
sources = []
for utt, gain in GAIN.items():
    s, fs = sf.read(f"samples/{utt}.flac", dtype="float32")   # 16 kHz
    sources.append(resample_poly(s * gain, 8000, fs))
n = min(map(len, sources))
mixture = sum(s[:n] for s in sources)

Because both sources are here, the separation can be scored rather than only listened to: on this mixture the model takes SI-SNR from 3.24 dB to 16.21 dB for one speaker and from -3.28 dB to 12.88 dB for the other.

The audio is from the LibriSpeech ASR corpus, (c) 2014 Vassil Panayotov, CC BY 4.0. The noisy version of Libri2Mix adds WHAM! noise, which is CC BY-NC 4.0 and is not included here.

Example ESPnet2 ENH model

anogkongda/librimix_enh_train_raw_valid.si_snr.ave

♻️ Imported from https://zenodo.org/record/4480771/

This model was trained by anogkongda using librimix/enh1 recipe in espnet.

Demo: How to use in ESPnet2

# coming soon

Citing ESPnet

@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson {Enrique Yalta Soplin} and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}
@inproceedings{hayashi2020espnet,
  title={{Espnet-TTS}: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit},
  author={Hayashi, Tomoki and Yamamoto, Ryuichi and Inoue, Katsuki and Yoshimura, Takenori and Watanabe, Shinji and Toda, Tomoki and Takeda, Kazuya and Zhang, Yu and Tan, Xu},
  booktitle={Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={7654--7658},
  year={2020},
  organization={IEEE}
}

or arXiv:

@misc{watanabe2018espnet,
      title={ESPnet: End-to-End Speech Processing Toolkit}, 
      author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Enrique Yalta Soplin and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
      year={2018},
      eprint={1804.00015},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}