figure-skating-cnn-bilstm

Figure Skating CNN-BiLSTM Action Classifier

A CNN-BiLSTM baseline for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series. Conv1D residual backbone for local temporal feature extraction, followed by a bidirectional LSTM for long-range temporal modeling.

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

Not transformers-compatible — this is a plain PyTorch nn.Module + state_dict checkpoint. Model source code (SkatingBiLSTMClassifier in the project's model_bilstm.py, which depends on model.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-cnn-bilstm"
ckpt_path = hf_hub_download(repo_id, "model_bilstm_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingBiLSTMClassifier (model_bilstm.py):
# model = SkatingBiLSTMClassifier(ckpt["in_features"], ckpt["num_classes"])
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_bilstm_fine.pt for model_bilstm_coarse.pt to load the 11-class variant.

Config

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Stopped epoch 51 (best @ ep31, val macro-F1 0.638)

Overall: accuracy 0.8689 · precision (macro/weighted) 0.5072 / 0.8800 · recall 0.5179 / 0.8689 · F1 0.5047 / 0.8697

Coarse (11-class)

Stopped epoch 45 (best @ ep25, val macro-F1 0.857)

Overall: accuracy 0.9089 · precision (macro/weighted) 0.8557 / 0.9126 · recall 0.8506 / 0.9089 · F1 0.8506 / 0.9089

Notable failure modes (from confusion matrix analysis)

• Flip ↔ Lutz is the dominant confusion (edge jumps — genuinely hard to distinguish from skeleton motion alone, mirrors real judging difficulty).
• Fine-grained macro-F1 is dragged down almost entirely by 1-2-sample tail classes (4Flip, 4Lutz, 4Loop, 1Salchow, 1Toeloop, 1Flip, 1Lutz) — a data scarcity issue, not an architecture weakness.
• Spins (CamelSpin, SitSpin, UprightSpin) and Sequence are solved territory: F1 ≥ 0.92 in both label spaces.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.