Downloads · 30 days
11
31% of all-time downloads
shahdelgouhary/asl-fingerspelling-transformer
asl-fingerspelling-transformer is a machine learning model from shahdelgouhary. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for keras. The card lists the license as mit.
A Transformer-based model that reads a sequence of hand/pose landmarks (extracted with MediaPipe) from a video of American Sign Language fingerspelling and predicts the spelled-out phrase, character by character.
Downloads · 30 days
11
31% of all-time downloads
All-time downloads
35
Public
Repo size
20.9 MB
Likes
0
Public
Click a slice to open those files.
.h520.9 MB · 100%
From the Hugging Face model README
A Transformer-based model that reads a sequence of hand/pose landmarks (extracted with MediaPipe) from a video of American Sign Language fingerspelling and predicts the spelled-out phrase, character by character.
Trained on the ASL Fingerspelling Kaggle dataset. Architecture follows the Keras "Automatic Speech Recognition with Transformer" example, adapted for landmark-sequence input instead of audio spectrograms.
FEATURE_COLUMNS
in modeling.py).Reverse-engineered from the actual weight shapes (see config.json):
num_hid=200, num_head=4, num_feed_forward=400, num_layers_enc=2,
num_layers_dec=1, num_classes=62, target_maxlen=64.
| File | Purpose |
|---|---|
modeling.py | Model architecture (must be imported to rebuild the model before loading weights) |
config.json | Hyperparameters used to build the architecture |
transformer_weights.h5 | Trained weights (model.save_weights(...) output — not a full SavedModel) |
inference.py | End-to-end example: landmarks in, predicted text out |
requirements.txt | Python dependencies |
⚠️ This model was saved with
save_weights(), notmodel.save(), so the weights file alone is not enough — you needmodeling.pyto reconstruct the exact architecture first, then load the weights into it.
from modeling import build_model, pre_process
import tensorflow as tf
model = build_model()
model.load_weights("transformer_weights.h5")
# landmarks: np.ndarray of shape (num_frames, num_feature_columns)
x = pre_process(tf.constant(landmarks, dtype=tf.float32))[None, ...]
token_ids = model.generate(x, target_start_token_idx=60)
See inference.py for the full pipeline including turning token ids back
into characters.
You'll need your character_to_prediction_index.json (character ↔ id
mapping) from training — it is not included in this repo. Place it next to
inference.py before running predictions.
irohith and shlomoron on Kaggle.