Downloads · 30 days
0
ialmajai/icefall-vsr-grid-conformer-ctc2-bpe-58-2026-04-29
icefall-vsr-grid-conformer-ctc2-bpe-58-2026-04-29 is a machine learning model from ialmajai. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Superseded (October 2026). The current system (phone lexicon, mean-face ROIs, AV-HuBERT layer 8; 7.14% WER on GRID unseen speakers and 8.39% zero-shot on Lombard GRID) is released at icefall-vsr-grid-conformer-ctc2-ph…
Downloads · 30 days
0
Access
Public
Updated Oct 3, 2026
Repo size
35.4 MB
Likes
0
Public
Click a slice to open those files.
.pt34.9 MB · 98%
From the Hugging Face model README
Superseded (October 2026). The current system (phone lexicon, mean-face ROIs, AV-HuBERT layer 8; 7.14% WER on GRID unseen speakers and 8.39% zero-shot on Lombard GRID) is released at icefall-vsr-grid-conformer-ctc2-phone-l8-2026-10-03. This repository remains the April 2026 BPE-58 release (6.72% WER) that is merged upstream in icefall.
This model is a reworked CTC-attention architecture trained using the codebase at github.com/k2-fsa/icefall/tree/master/egs/grid/VSR/conformer_ctc2.
Mouth regions of interest (ROIs) are tracked and extracted at 64x64 pixels using Dlib, then upsampled to 88x88 pixels to meet AV-HuBERT's input requirements. Features are extracted from layer 9 of the pretrained AV-HuBERT model (base_vox_iter5.pt) and projected to 256 dimensions.
The convolution/subsampling stage is omitted, preserving visual features at their native 25 fps temporal resolution. The encoder consists of 4 reworked Conformer layers and 2 reworked Transformer decoder layers.
The model is evaluated on the unseen speaker task. Speakers S1, S2, S20, and S22 are held out exclusively for testing and are not seen during training.
The model was trained for 30 epochs. Convergence is notably faster than the original architecture, so extended training was not required.
https://demo.ibrahimalmajai.com, https://www.linkedin.com/feed/update/urn:li:share:7482828720160935936
For detailed information, please refer to https://github.com/k2-fsa/icefall/blob/master/egs/grid/VSR/RESULTS.md or https://github.com/k2-fsa/icefall/pull/2081.