vit_lstm_jester

Hand Gesture Recognition – Vision Transformer + LSTM

Model Description

This repository contains a Bidirectional LSTM model for dynamic hand gesture recognition, trained on features extracted from the Jester dataset. The model is designed to classify temporal sequences of frame-level embeddings.

The overall architecture is a two-stage process:

1. Feature Extraction: A pre-trained Vision Transformer (ViT-B/16), fine-tuned on the HaGRID dataset, is used as a frozen feature extractor. It processes each video frame (224x224) and generates a 768-dimensional embedding. No training was performed on the ViT in this project; it is used as-is.
2. Temporal Classification: The sequence of frame embeddings is then fed into the Bidirectional LSTM model, which learns to recognize the temporal patterns of different hand gestures.

Model Version: clean-baseline-v2.0-hf Architecture: Bidirectional LSTM with ViT-B/16 feature extractor Parameters (LSTM): ~514k Framework: TensorFlow + Keras (for LSTM), PyTorch + timm (for ViT feature extraction)

Performance Metrics (LSTM Classifier)

Training Details

Dataset

Name: 20bn-Jester Dataset Source: local Train samples: 50,420 Validation samples: 7,047

Training Configuration (LSTM)

Batch Size: 8 Optimizer: Adam Loss: Categorical Cross-Entropy Learning Rate: 0.001 (reduced by ReduceLROnPlateau) Epochs Trained: Up to 30 (best weights restored from epoch 29 based on validation accuracy) Regularization: Dropout (0.4, 0.3), L2 (1e-4) on Dense layer Callbacks: ModelCheckpoint, ReduceLROnPlateau, EarlyStopping (patience=7)

Data Processing

Preprocessing: Single deterministic pipeline (no augmentation) Resize: 224×224 Normalization: ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) Temporal Sampling: Fixed sequence length of 37 frames Padding: Zero-vector with masking Augmentation: none (clean baseline principle) Test-Time Augmentation: False (disabled)

Architecture Details

Type: standard ViT-B/16 (no modifications, frozen during LSTM training) ViT Output: 768-D feature embeddings per frame Classification Head (ViT): Removed for feature extraction LSTM Model: - Input: Sequences of ViT embeddings (37 frames, 768 dimensions) - Layers: Masking, Bidirectional LSTM (256 units, return_sequences=True), Dropout (0.4), Bidirectional LSTM (128 units), Dropout (0.4), Dense (128 units, ReLU, L2 regularization), Dropout (0.3), Dense (27 classes, Softmax) Total Trainable Parameters (LSTM): ~514k Dropout: 0.4, 0.3

Gesture Classes (27 total)

'Doing other things', 'Drumming Fingers', 'No gesture', 'Pulling Hand In', 'Pulling Two Fingers In', 'Pushing Hand Away', 'Pushing Two Fingers Away', 'Rolling Hand Backward', 'Rolling Hand Forward', 'Shaking Hand', 'Sliding Two Fingers Down', 'Sliding Two Fingers Left', 'Sliding Two Fingers Right', 'Sliding Two Fingers Up', 'Stop Sign', 'Swiping Down', 'Swiping Left', 'Swiping Right', 'Swiping Up', 'Thumb Down', 'Thumb Up', 'Turning Hand Clockwise', 'Turning Hand Counterclockwise', 'Zooming In With Full Hand', 'Zooming In With Two Fingers', 'Zooming Out With Full Hand', 'Zooming Out With Two Fingers'

Usage

Refer to how_to_use.py for a complete end-to-end example.

Limitations

• Requires consistent frame rate and sampling • Sensitive to heavy occlusion and motion blur • Assumes a single dominant gesture per clip • Performance depends on ViT embedding quality

License

MIT License

Dataset Acknowledgment

This model was trained on the Jester Dataset, a large-scale video dataset for hand gesture recognition. We would like to thank Twenty Billion Neurons (TwentyBN) for creating and sharing this dataset.

• Dataset Homepage: Jester Dataset
• License: The Jester dataset is available under a Creative Commons license. Please refer to the dataset's official website for more information on its license.

How to Use

This model consists of two parts: a Vision Transformer (ViT) feature extractor and an LSTM-based temporal classifier.

1. Installation

First, make sure you have the required libraries installed:

pip install tensorflow torch torchvision timm transformers

2. Loading the Models

The ViT backbone can be loaded from the Hugging Face Hub here, while the trained LSTM model can be loaded from the best_lstm_model.keras file in this repository.