PG-XFormer

PG-XFormer

Pose-guided cross-modal exercise recognition with a safety-gated feedback layer.

PG-XFormer fuses a CTR-GCN skeleton stream and a VideoMAE video stream with symmetric cross-attention to recognise physiotherapy exercises from a short clip and its 2D pose. It is the recognition backbone of the paper "PG-XFormer: Pose-Guided Cross-Modal Exercise Monitoring with Agentic Feedback". This repository holds the trained weights for every configuration and seed reported in the paper; code, split definitions, per-clip predictions, and the feedback audit are in PG-XFormer-artifacts.

All PG-XFormer resources are grouped in the PG-XFormer collection.

Model details

Checkpoints

model.safetensors is the default checkpoint (cross-attention, seed 0; fixed in advance as the initialisation of the clinical experiments, not chosen by test score). checkpoints/synthetic/<run>.safetensors holds the other runs, named <fusion>_<objective>_s<seed> (ce: cross-entropy; align: + InfoNCE, λ = 0.1). Held-out test results (70 clips, seven per exercise):

Runs without a listed checkpoint (e.g. xattn_ce_s1) were trained before checkpoint saving was enabled; their per-clip predictions are in the artifacts repository.

Usage

pip install "torch>=2.4" "transformers>=5" safetensors huggingface_hub numpy

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("pgx", hf_hub_download("FatimahEmadEldin/PG-XFormer", "pgxformer_load.py"))
pgx = importlib.util.module_from_spec(spec); spec.loader.exec_module(pgx)

model, E = pgx.load_pgxformer("xattn_ce_s0")      # or e.g. "concat_ce_s2"
logits = model(video, keypoints)                  # video: (B,16,3,224,224); keypoints: (B,16,17,2)

keypoints must be normalised with E.normalise_keypoints. The checkpoints use the transformers>=5 VideoMAE parameter naming.

Training

• Data: synthetic InfiniteRep renders (archive), 700 clips of ten exercises; instance-disjoint, class-stratified split of 560 / 70 / 70 clips (split files in the artifacts repository). Keypoints were estimated from the RGB frames with an off-the-shelf 2D pose estimator.
• Procedure: AdamW with cosine schedule, 20 epochs, effective batch 8 (4 × 2 accumulation), fp16 autocast; fusion learning rate 1e-4 (concatenation 3e-4); checkpoint selected on the development split; three seeds per configuration.

Evaluation (mean ± s.d. over three seeds)

Fusion is significantly better than the skeleton stream (McNemar p < 1e-18). Differences between the fusion variants and video alone are not statistically significant on the 70-clip test set; cross-attention is significantly better calibrated than bottleneck fusion. Adding the InfoNCE term did not help (98.5 → 96.2%).

Clinical transfer (KIMORE). With a leakage-free five-fold protocol, synthetic pretraining gave no benefit over KIMORE-only training after fine-tuning (93.1 vs. 93.7% macro-F1), although frozen synthetic features reached 72.4% under a linear probe (chance 20%). KIMORE-adapted weights are not released because KIMORE is distributed by its authors on request.

Feedback layer

In the paper the recogniser is paired with a kinematic error report and a language-model cue generator enclosed by a rejection gate (clean and critical reports answered deterministically) and an output verifier (cues naming unreported body regions are replaced). The gate removed unrequested advice on clean repetitions (73.4% → 0% for Qwen3-235B) and made Qwen3-0.6B / 1.7B equally safe. Its code and every generated cue are in the artifacts repository.

Intended use and limitations

• Research on exercise recognition and safe automated feedback. Not a medical device and not validated for clinical decision-making; no clinician has rated the feedback.
• Trained only on synthetic renders of ten exercises; accuracy on real footage, other exercises, or other camera set-ups is unknown. The 70-clip test set limits statistical power.
• Keypoint quality matters: the skeleton stream is weak on estimated synthetic keypoints.

Licence

CC BY-NC 4.0, inherited from the VideoMAE-base weights used for initialisation. InfiniteRep is a synthetic dataset by Infinity AI (research use with attribution).

Citation

Citation to be added on publication. Please also cite InfiniteRep, KIMORE (Capecci et al., IEEE TNSRE 2019), CTR-GCN (Chen et al., ICCV 2021), and VideoMAE (Tong et al., NeurIPS 2022).