asl-citizen-small-transformer-encoder-200

ASL Citizen Small Transformer Isolated Sign Encoder

This repository contains the trained Small Transformer Encoder model for isolated American Sign Language recognition.

The model was trained as a comparison baseline for the FYP project:

Bidirectional ASL ↔ English Translation System

The goal of this experiment was to compare a self-attention-based temporal encoder against the previously trained BiGRU + Attention baseline.

Project Stage

ASL Citizen processed keypoints-200
        ↓
Small Transformer isolated sign recognition model
        ↓
Comparison with BiGRU + Attention baseline
        ↓
Select stronger encoder for future How2Sign CTC stage

Dataset

Training dataset:

SharoonArshad/asl-citizen-processed-200

The dataset uses the processed ASL Citizen keypoints-200 format:

(N, 200, 450)

Where:

200 = fixed sequence length
450 = 225 position features + 225 velocity features
225 = 75 landmarks × 3 coordinates
75 landmarks = 33 pose + 21 left hand + 21 right hand

Model Architecture

The model uses a small Transformer encoder over temporal keypoint sequences:

Input:                  (B, 200, 450)
Input LayerNorm:        450
Linear Projection:      450 → 256
CLS Token:              prepended to the sequence
Positional Encoding:    sinusoidal
Transformer Encoder:    3 layers
Attention Heads:        4
Feedforward Dimension:  768
Pooling:                CLS token
Classifier Head:        200 ASL classes

Training Configuration

Main training settings:

Batch size: 32
Max epochs: 60
Best epoch: 52
Early stopping patience: 10
Learning rate: 0.0002
Weight decay: 0.0001
Dropout: 0.3
Label smoothing: 0.1
Gradient clipping: 1.0
Scheduler: cosine
Optimizer: AdamW
Mixed precision: enabled on GPU

Results

Small Transformer final results:

Best epoch: 52
Best validation Top-1 accuracy: 78.13%
Test loss: 1.3240
Test correct: 3238/3552
Test Top-1 accuracy: 91.16%
Test Top-5 accuracy: 99.13%
Test Macro F1: 0.9118
Test Micro F1: 0.9119
Test Weighted F1: 0.9104
Mean per-class accuracy: 0.9144
Training time: 06m 40s

Comparison with BiGRU + Attention Baseline

The BiGRU + Attention model achieved stronger performance on the same ASL Citizen processed dataset:

Metric                  BiGRU + Attention      Small Transformer
Validation Top-1        85.47%               78.13%
Test Top-1              98.48%               91.16%
Test Top-5              99.58%               99.13%
Test Macro F1           0.9850               0.9118
Test Weighted F1        0.9848               0.9104
Training Time           26m 25s              06m 40s

Interpretation

The Small Transformer trained successfully, but it underperformed the BiGRU + Attention model.

The best validation Top-1 accuracy was:

BiGRU + Attention: 85.47%
Small Transformer: 78.13%

Therefore, the BiGRU + Attention encoder was selected as the stronger isolated sign encoder for future transfer into the continuous How2Sign CTC stage.

The Small Transformer is still useful as a comparison baseline for the thesis because it demonstrates that a self-attention temporal encoder was evaluated against a recurrent attention-based encoder.

Repository Files

asl_isolated_small_transformer.zip
checkpoints/best_small_transformer_model.pt
checkpoints/best_transformer_encoder_only.pt
checkpoints/last_small_transformer_model.pt
reports/final_metrics.json
reports/training_history.csv
reports/test_classification_report.csv
reports/test_classification_report.json
reports/test_confusion_matrix.npy
reports/test_per_class_accuracy.csv
metadata/label_to_id.json
metadata/id_to_label.json
training_config.json

Important Files

Full model checkpoint

checkpoints/best_small_transformer_model.pt

Use this file if you want to reload the complete Small Transformer classifier for isolated sign prediction.

Encoder-only checkpoint

checkpoints/best_transformer_encoder_only.pt

Use this file if you want to reuse the Transformer encoder in future experiments.

Full ZIP archive

asl_isolated_small_transformer.zip

This archive contains the full training output folder, including checkpoints, reports, label maps, and configuration files.

Loading the Model in Kaggle

Example download code:

from huggingface_hub import hf_hub_download

repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"

model_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="checkpoints/best_small_transformer_model.pt"
)

encoder_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="checkpoints/best_transformer_encoder_only.pt"
)

print(model_path)
print(encoder_path)

Loading the Full ZIP Archive

from huggingface_hub import hf_hub_download
import zipfile
from pathlib import Path

repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"

zip_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="asl_isolated_small_transformer.zip"
)

extract_dir = Path("/kaggle/working/asl_isolated_small_transformer_loaded")
extract_dir.mkdir(parents=True, exist_ok=True)

with zipfile.ZipFile(zip_path, "r") as zip_ref:
    zip_ref.extractall(extract_dir)

print("Extracted to:", extract_dir)

Thesis Use

This model can be documented as the Transformer-based isolated sign recognition baseline.

Suggested thesis statement:

A Small Transformer Encoder was trained on the processed ASL Citizen keypoints-200 dataset for 200 isolated ASL classes. The model achieved 78.13% validation Top-1 accuracy and 91.16% test Top-1 accuracy. Although it trained faster than the BiGRU + Attention model, it achieved lower validation and test performance. Therefore, the BiGRU + Attention encoder was selected as the primary isolated sign encoder for future CTC-based continuous sign recognition.

Limitations

• The model is trained on isolated signs, not continuous signing.
• It recognizes 200 selected ASL classes from the processed ASL Citizen dataset.
• It underperformed the BiGRU + Attention baseline on validation accuracy and test accuracy.
• It does not yet perform ASL sentence translation.
• A larger Transformer may require more data or stronger regularization.

Next Step

The next project stage is to use the stronger BiGRU + Attention encoder checkpoint as the pretrained visual encoder for a CTC-based How2Sign continuous sign recognition model.