figure-skating-transformer-bilstm

Figure Skating Transformer-BiLSTM Action Classifier

A two-stage temporal model for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series, with no conv backbone: a linear projection + sinusoidal positional encoding + nn.TransformerEncoder (self-attention, global context) feeding a bidirectional LSTM decoder (re-imposes local temporal order before pooling).

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

Not transformers-compatible in the HF sense — this is a plain PyTorch nn.Module + state_dict checkpoint (custom architecture, not a Hub AutoModel). Model source code (SkatingTransformerBiLSTMClassifier in the project's model_transformer_bilstm.py, which depends on model.py and model_transformers.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-transformer-bilstm"
ckpt_path = hf_hub_download(repo_id, "model_transformer_bilstm_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingTransformerBiLSTMClassifier (model_transformer_bilstm.py):
# model = SkatingTransformerBiLSTMClassifier(ckpt["in_features"], ckpt["num_classes"])
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_transformer_bilstm_fine.pt for model_transformer_bilstm_coarse.pt to load the 11-class variant.

Config

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Early-stopped epoch 64 (best val macro-F1 0.449) — notably unstable val loss/F1 trajectory during training.

Overall: accuracy 0.7297 · precision (macro/weighted) 0.3764 / 0.7598 · recall 0.4114 / 0.7297 · F1 0.3615 / 0.7341

Coarse (11-class)

Early-stopped epoch 34 (best val macro-F1 0.793)

Overall: accuracy 0.8679 · precision (macro/weighted) 0.7996 / 0.8741 · recall 0.8081 / 0.8679 · F1 0.7995 / 0.8683

Notable failure modes (from confusion matrix analysis, coarse label space)

• Weakest architecture of the family, especially fine-grained — 73.0% acc / 0.361 macro-F1 is the worst fine-grained result measured across every architecture tried; early-stops fastest on coarse (epoch 34) too.
• Flip → Lutz collapse is severe: 18/47 Flip test samples (38%) predicted as Lutz — the single worst individual confusion measured across every architecture and label space tested.
• Unique failure mode not seen elsewhere: UprightSpin → Sequence leakage (8/108, 7.4%).
• No conv backbone to establish local temporal structure before self-attention appears to hurt more than it helps at this data scale.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.