Downloads · 30 days
35
9% of all-time downloads
japanese-asr/en-cascaded-s2t-translation
en-cascaded-s2t-translation is a automatic speech recognition model from japanese-asr. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as apache-2.0.
This is a pipeline for speech-to-text translation from English speech to any target language text based on the cascaded approach, that consists of ASR and translation. The pipeline employs distil-whisper/distil-large-…
Downloads · 30 days
35
9% of all-time downloads
All-time downloads
406
Public
Parameters
756M
3 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors3 GB · 100%
From the Hugging Face model README
This is a pipeline for speech-to-text translation from English speech to any target language text based on the cascaded approach, that consists of ASR and translation. The pipeline employs distil-whisper/distil-large-v3 for ASR (English speech -> English text) and facebook/nllb-200-3.3B for text translation. The input must be English speech, while the translation can be in any languages NLLB trained on. Please find the all available languages and their language codes here.
Model for Japanese speech translation is available at ja-cascaded-s2t-translation.
The folloiwng table shows CER computed over the reference and predicted translation for translating English speech to Japanese text task (subsets of CoVoST2 and Fleurs) with different size of NLLB along with OpenAI Whisper models.
| model | CoVoST2 (En->Ja) | Fleurs (En->JA) |
|---|---|---|
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-3.3B) | 62.4 | 63.5 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-1.3B) | 64.4 | 67.2 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-distilled-1.3B) | 62.4 | 62.9 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-distilled-600M) | 63.4 | 66.2 |
| openai/whisper-large-v3 | 178.9 | 209.5 |
| openai/whisper-large-v2 | 179.6 | 201.8 |
| openai/whisper-large | 178.7 | 201.8 |
| openai/whisper-medium | 178.7 | 202 |
| openai/whisper-small | 178.9 | 206.8 |
| openai/whisper-base | 179.5 | 214.2 |
| openai/whisper-tiny | 185.2 | 200.5 |
See https://github.com/kotoba-tech/kotoba-whisper for the evaluation detail.
Due to the nature of cascaded approach, the pipeline has additional complexity compared to the single end2end OpenAI whisper models for the sake of high accuracy. Following table shows the mean inference time in second averaged over 10 trials on audio sample with different durations.
| model | 10 | 30 | 60 | 300 |
|---|---|---|---|---|
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-3.3B) | 0.173 | 0.247 | 0.352 | 1.772 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-1.3B) | 0.173 | 0.24 | 0.348 | 1.515 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-distilled-1.3B) | 0.17 | 0.245 | 0.348 | 1.882 |
| japanese-asr/en-cascaded-s2t-translation (facebook/nllb-200-distilled-600M) | 0.108 | 0.179 | 0.283 | 1.33 |
| openai/whisper-large-v3 | 0.061 | 0.184 | 0.372 | 1.804 |
| openai/whisper-large-v2 | 0.062 | 0.199 | 0.415 | 1.854 |
| openai/whisper-large | 0.062 | 0.183 | 0.363 | 1.899 |
| openai/whisper-medium | 0.045 | 0.132 | 0.266 | 1.368 |
| openai/whisper-small | 0.135 | 0.376 | 0.631 | 3.495 |
| openai/whisper-base | 0.054 | 0.108 | 0.231 | 1.019 |
| openai/whisper-tiny | 0.045 | 0.124 | 0.208 | 0.838 |
Here is an example to translate English speech into Japanese text translation. First, download a sample speech.
wget https://huggingface.co/datasets/japanese-asr/en_asr.esb_eval/resolve/main/sample.wav -O sample_en.wav
Then, run the pipeline as below.
from transformers import pipeline
# load model
pipe = pipeline(
model="japanese-asr/en-cascaded-s2t-translation",
model_translation="facebook/nllb-200-distilled-600M",
tgt_lang="jpn_Jpan",
model_kwargs={"attn_implementation": "sdpa"},
chunk_length_s=15,
trust_remote_code=True,
)
# translate
output = pipe("./sample.wav")
Other NLLB models can be used by setting model_translation such as following.