dllm-vsr

Diffusion Large Language Models for Visual Speech Recognition

Paper checkpoints for DLLM-VSR — adapting the Dream-7B discrete-diffusion LLM to Visual Speech Recognition (VSR) on LRS3.

• Paper: arxiv.org/abs/2605.28456
• Code: github.com/JeongHun0716/dllm-vsr
• Authors: Jeong Hun Yeo, Chae Won Kim, Hyeongseop Rha, Yong Man Ro

Contents

Each dream_stage2/ holds trainable_model.safetensors (LoRA adapters + visual-feature projector). Each len_pred/ holds trainable_model.pt (small Transformer over visual features) together with its precomputed outputs len_pred_test.jsonl (LRS3 test) and len_pred_wildvsr.jsonl (WildVSR) — the K_pred ± 5 candidate pools used in the paper, so the evaluation scripts can skip length-predictor inference.

Note: Visual encoder weights (USR 2.0 Huge, AV-HuBERT Large) are not redistributed here. Download them from the original repos:

• AV-HuBERT: https://github.com/facebookresearch/av_hubert
• USR 2.0: https://github.com/ahaliassos/usr2 (Huge, LRS2+LRS3+Vox2+AVS pretrain checkpoint)

Results (WER, %)

All entries are trained on LRS3 (433h) only.

LRS3 test:

WildVSR (out-of-domain, same LRS3-trained models):

Usage

huggingface-cli download jh-y/dllm-vsr --local-dir ckpt

Then follow the code repo's README for environment setup, preprocessing (auto-avsr pipeline), and inference scripts.

Citation

@article{yeo2026dllmvsr,
  title={Diffusion Large Language Models for Visual Speech Recognition},
  author={Yeo, Jeong Hun and Kim, Chae Won and Rha, Hyeongseop and Ro, Yong Man},
  journal={arXiv preprint arXiv:2605.28456},
  year={2026}
}