avista-base-v2

<div align="center"> <video width="80%" controls> <source src="https://huggingface.co/datasets/enactic/assets/resolve/main/AVista%20demo.mp4" type="video/mp4"> Your browser does not support the video tag. </video> </div>

AVista Base v2 🐦‍🔥

This is AVHuBERT (Audio-Visual Hidden Unit BERT) Base model for AVSR (Audio-Visual Speech Recognition) task, derived from enactic/japanese-avhubert-base.

This model is fine-tuned on approximately 1,300h of Japanese audio-visual dataset combined with 35,000h of Japanese audio-only dataset, ReazonSpeech.

Usage

Please install dependencies first.

$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/avsr

Using transformers directly

You can load AVSR models by directly using Hugging Face transformers if you trust our remote code.

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

processor = AutoProcessor.from_pretrained("enactic/avista-base-v2", trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained("enactic/avista-base-v2", trust_remote_code=True)

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)

Using reazonspeech.avsr package

You can also load AVSR models by using reazonspeech.avsr. If you don't want to use remote code for security reasons for example, you can use the following code.

from reazonspeech.avsr import AVHubertProcessor, AVHubertForConditionalGeneration

processor = AVHubertProcessor.from_pretrained("enactic/avista-base-v2")
model = AVHubertForConditionalGeneration.from_pretrained("enactic/avista-base-v2")

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)

Test Results

We report the Character Error Rate (CER) on an out-of-domain evaluation dataset that was internally collected for AVSR benchmarking.

The following table presents the benchmark results of this model and Japanese ASR models under different noise levels and noise types.

Details of the dataset and the complete benchmark results can be found here.

+ ReazonSpeech Speech

+ JSUT Speech

+ Babble

+ Music

+ Environmental Noise

Citation

@misc{enactic/avista-base-v2,
  title={avista-base-v2},
  author={Sasaki, Yuta},
  url = {https://huggingface.co/enactic/avista-base-v2},
  year = {2025}
}

@article{shi2022avhubert,
    author  = {Bowen Shi and Wei-Ning Hsu and Kushal Lakhotia and Abdelrahman Mohamed},
    title = {Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction},
    journal = {arXiv preprint arXiv:2201.02184}
    year = {2022}
}

@article{shi2022avsr,
    author  = {Bowen Shi and Wei-Ning Hsu and Abdelrahman Mohamed},
    title = {Robust Self-Supervised Audio-Visual Speech Recognition},
    journal = {arXiv preprint arXiv:2201.01763}
    year = {2022}
}

License

Apache License 2.0