figure-skating-transformer

Figure Skating Transformer Action Classifier

A Conv1D backbone + Transformer-encoder baseline for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series. Sinusoidal positional encoding + a real 3-layer nn.TransformerEncoder (self-attn + FFN per layer) on top of the same Conv1D residual backbone used across this model family.

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

Not transformers-compatible in the HF sense — this is a plain PyTorch nn.Module + state_dict checkpoint (custom architecture, not a Hub AutoModel). Model source code (SkatingActionClassifier in the project's model_transformers.py, which depends on model.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-transformer"
ckpt_path = hf_hub_download(repo_id, "model_transformer_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingActionClassifier (model_transformers.py):
# model = SkatingActionClassifier(ckpt["in_features"], ckpt["num_classes"])
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_transformer_fine.pt for model_transformer_coarse.pt to load the 11-class variant.

Config

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Early-stopped epoch 42 (best val macro-F1 0.572)

Overall: accuracy 0.8348 · precision (macro/weighted) 0.4511 / 0.8364 · recall 0.5129 / 0.8348 · F1 0.4642 / 0.8287

Coarse (11-class)

Early-stopped epoch 61 (best val macro-F1 0.860)

Overall: accuracy 0.8989 · precision (macro/weighted) 0.8390 / 0.8989 · recall 0.8158 / 0.8989 · F1 0.8233 / 0.8970

Notable failure modes (from confusion matrix analysis, coarse label space)

• Best Axel (95.1%) and Toeloop (87.0%) recall of all architectures tested, but by far the worst Salchow recall (46.7%) — heavily biased toward calling Salchow "Toeloop" (11/30 test samples, the single worst confusion pair measured across every model tried).
• Flip ↔ Lutz confusion present but less severe than in other architectures.
• Fine-grained macro-F1 is dragged down almost entirely by 1-2-sample tail classes (4Flip, 4Lutz, 4Loop, 1Salchow, 1Toeloop, 1Flip, 1Lutz) — a data scarcity issue, not an architecture weakness.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.