avista-large

<div align="center"> <video width="80%" controls> <source src="https://huggingface.co/datasets/enactic/assets/resolve/main/AVista%20demo.mp4" type="video/mp4"> Your browser does not support the video tag. </video> </div>

AVista Large 🐦‍🔥

This is AVHuBERT (Audio-Visual Hidden Unit BERT) Large model for AVSR (Audio-Visual Speech Recognition) task, derived from enactic/japanese-avhubert-large.

This model is fine-tuned on approximately 1,300h of Japanese audio-visual dataset.

Usage

Using transformers directly

Please install the following requirements to build the model by using transformers directly.

$ pip install einops \
              librosa \
              opencv-python \
              python_speech_features \
              soundfile
              torch\
              torchvision\
              transformers==4.44.1 \
              mediapipe \
              deepspeed

You can load AVSR models by directly using Hugging Face transformers if you trust our remote code.

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

processor = AutoProcessor.from_pretrained("enactic/avista-large", trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained("enactic/avista-large", trust_remote_code=True)

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)

Using reazonspeech.avsr package

You can also load AVSR models by using reazonspeech.avsr. If you don't want to use remote code for security reasons for example, you can use the following code.

$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/avsr

from reazonspeech.avsr import AVHubertProcessor, AVHubertForConditionalGeneration

processor = AVHubertProcessor.from_pretrained("enactic/avista-large")
model = AVHubertForConditionalGeneration.from_pretrained("enactic/avista-large")

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)

Test Results

We report the Character Error Rate (CER) on an out-of-domain evaluation dataset that was internally collected for AVSR benchmarking.

The following table presents the benchmark results of this model and Japanese ASR models under different noise levels and noise types.

Details of the dataset and the complete benchmark results can be found here.

+ ReazonSpeech Speech

+ JSUT Speech

+ Babble

+ Music

+ Environmental Noise

Citation

@misc{enactic/avista-large,
  title={avista-large},
  author={Sasaki, Yuta},
  url = {https://huggingface.co/enactic/avista-large},
  year = {2025}
}

@article{shi2022avhubert,
    author  = {Bowen Shi and Wei-Ning Hsu and Kushal Lakhotia and Abdelrahman Mohamed},
    title = {Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction},
    journal = {arXiv preprint arXiv:2201.02184}
    year = {2022}
}

@article{shi2022avsr,
    author  = {Bowen Shi and Wei-Ning Hsu and Abdelrahman Mohamed},
    title = {Robust Self-Supervised Audio-Visual Speech Recognition},
    journal = {arXiv preprint arXiv:2201.01763}
    year = {2022}
}

License

Apache License 2.0