owsm_v3.1_ebf_small_lowrestriction

Usage

import librosa
from espnet2.bin.s2t_inference import Speech2Text

s2t = Speech2Text.from_pretrained(
    model_tag="espnet/owsm_v3.1_ebf_small_lowrestriction", lang_sym="<eng>", task_sym="<asr>", beam_size=5
)
# OWSM is trained on 16 kHz; each call decodes 30 s, padded or trimmed
speech, rate = librosa.load("audio.wav", sr=16000, mono=True)
text, token, token_int, text_nospecial, hyp = s2t(speech)[0]
print(text_nospecial)  # `text` keeps OWSM's own <eng><asr> markers
# for a recording longer than 30 s: s2t.decode_long(speech) -> (start, end, text)

OWSM: Open Whisper-style Speech Model

OWSM is an Open Whisper-style Speech Model from CMU WAVLab. It reproduces Whisper-style training using publicly available data and an open-source toolkit ESPnet.

Our demo is available here. The project page contains various resources.

OWSM v3.1 is an improved version of OWSM v3. It significantly outperforms OWSM v3 in almost all evaluation benchmarks. We do not include any new training data. Instead, we utilize a state-of-the-art speech encoder, E-Branchformer.

This is a small size model with 367M parameters and is trained on 70k hours of public speech data with lower restrictions (compared to the full OWSM data). Please check our project page for more information. Specifically, it supports the following speech-to-text tasks:

• Speech recognition
• Utterance-level alignment
• Long-form transcription
• Language identification

Citations

OWSM-CTC

@inproceedings{owsm-ctc,
    title = "{OWSM}-{CTC}: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification",
    author = "Peng, Yifan  and
      Sudo, Yui  and
      Shakeel, Muhammad  and
      Watanabe, Shinji",
    booktitle = "Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)",
    year = "2024",
    month= {8},
    url = "https://aclanthology.org/2024.acl-long.549",
}

OWSM v3.1 and v3.2

@inproceedings{owsm-v32,
  title={On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models},
  author={Jinchuan Tian and Yifan Peng and William Chen and Kwanghee Choi and Karen Livescu and Shinji Watanabe},
  booktitle={Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH)},
  year={2024},
  month={9},
  pdf="https://arxiv.org/pdf/2406.09282"
}
@inproceedings{owsm-v31,
  title={{OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer}},
  author={Yifan Peng and Jinchuan Tian and William Chen and Siddhant Arora and Brian Yan and Yui Sudo and Muhammad Shakeel and Kwanghee Choi and Jiatong Shi and Xuankai Chang and Jee-weon Jung and Shinji Watanabe},
  booktitle={Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH)},
  year={2024},
  month={9},
  pdf="https://arxiv.org/pdf/2401.16658",
}

Initial OWSM (v1, v2, v3)

@inproceedings{owsm,
  title={Reproducing Whisper-Style Training Using An Open-Source Toolkit And Publicly Available Data},
  author={Yifan Peng and Jinchuan Tian and Brian Yan and Dan Berrebbi and Xuankai Chang and Xinjian Li and Jiatong Shi and Siddhant Arora and William Chen and Roshan Sharma and Wangyou Zhang and Yui Sudo and Muhammad Shakeel and Jee-weon Jung and Soumi Maiti and Shinji Watanabe},
  booktitle={Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
  year={2023},
  month={12},
  pdf="https://arxiv.org/pdf/2309.13876",
}