models-vsro200

VSRo-200: Romanian Visual Speech Recognition Models

This repository hosts the encoder-decoder VSR model checkpoints introduced in the paper VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness.

The models are MultiVSR backbones fine-tuned on the VSRo-200 corpus, a 200-hour collection of Romanian podcast recordings. For training code, data preparation scripts, and inference instructions, please refer to the GitHub repository.

Checkpoints

All checkpoints follow the naming pattern model_[hours]_[type].pt:

• _annot — trained on human-annotated transcriptions
• _auto — trained on automatically generated pseudo-labels
• _shuffle — alternative data splits used for variance analysis (100h models)
• _males / _females / _mix — gender-controlled 40h subsets used for bias analysis

Results

All results are reported in Word Error Rate (WER, %) and Character Error Rate (CER, %) on the Test Unseen and Test Seen splits. Lower is better.

Human Annotated Data

Whisper Pseudo Labels

A variance analysis across three random shuffles of the 100h subsets yields a mean Word Error Rate (WER) of 53.21% (± 0.37) for the human-annotated data and 53.82% (± 0.17) for the auto-generated data.

Out-of-distribution robustness

• Test Seen / Unseen (In-Domain): Baseline performance on podcast data, tested on our 200h-model.
• Vlogs: Unconstrained videos shot in different camera angles, dynamic lighting, movement.
• Specific domains: Content featuring highly specialized or technical vocabulary (e.g., medical, scientific).
• Noisy: Videos with poor resolution, bad lighting, or heavy motion blur.
• Archival (Black & White): Historical footage with distinct visual artifacts, atypical framerates, and lack of color information.
• Global OOD: The aggregated metrics across all out-of-distribution subsets.

Metrics Note

• Duration: Each OOD category consists of 15 minutes of video content.
• OOV Token (%): The percentage of total words in the evaluation set that do not appear in the training data. Measures how often unknown words occur.
• OOV Type (%): The percentage of unique words in the evaluation set that do not appear in the training data. Measures the diversity of unknown words.

Gender bias analysis (40h models)

To evaluate gender bias and cross-speaker generalization, we trained 40-hour baseline models on male-only, female-only, and mixed datasets.

Test Unseen

Test Seen

Citation

If you use these models, please cite:

@inproceedings{vsro200,
  title  = {VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness},
  author = {...},
  year   = {...}
}