skeletonllm-ntu60-zs-55-5

SkeletonLLM (Stage 4) on NTU-60 zero-shot 55/5

Reproduction of the SkeletonLLM recipe from Universal Skeleton Understanding via Differentiable Rendering and MLLMs (Wang et al., ICML 2026, arXiv:2603.18003, code). The authors release renderer-only DrAction weights and no full checkpoint, so this is a full model trained from that starting point.

The model takes an NTU .skeleton file, renders it to 12 frames at 448x448 with DrAction, and answers a multiple-choice action question over the split's class list.

Result

75.22% on NTU-60 zero-shot 55/5 (1,380 test clips, 5 unseen classes, chance 20%).

274 of 276 writing clips are predicted as reading; the other four classes average 94.0%. The two are each other's top entry in the paper's own ntu60_similar_actions.json, which is the input to the Stage 2 discriminative finetuning that this model does not have.

For reference, the paper reports 87.37 for the full four-stage pipeline and 84.43 for its minimal Stage 1 + Stage 4 pipeline on this split.

How this differs from the paper

• Stage 1 (render warm-up) was not run. The renderer is the authors' released draction_candidate_01.safetensors, frozen. Training Stage 1 from base InternVL3-8B produces a renderer whose parameters are left uninitialised by from_pretrained fast-init, which yields NaN on every rendered frame.
• Stages 2 (Disc-FT) and 3 (CR-Distill) were not run. Both are optional in the authors' README.
• The NFM temporal GRU is disabled, matching the released candidate weights, which contain no GRU tensors.
• Stage 4 used 40% of the training split (14,704 of 36,758 clips, seeded and stratified by class) for 1 epoch instead of the full split for 3 epochs.

Everything else follows shell/train.sh: InternVL3-8B base, 448px, 12 frames, 10 line samples, NFM on, frozen LLM and vision tower, LoRA r=32 merged into these weights, ZeRO-1, bf16, lr 2e-5.

Usage

Evaluate with the authors' script:

python eval/test_ntu.py --model-path <this repo> --skeleton-root /path/to/ntu/raw --output-file out.tsv --split ntu60_55_cs --render-frames 12 --render-size 448 --enable-nfm --simple-single-gpu

License and provenance

Derived from OpenGVLab/InternVL3-8B; that model's license and the license of its Qwen2.5 language backbone apply. Trained on NTU RGB+D skeletons, which are distributed under the ROSE Lab academic release agreement; no dataset content is redistributed here, only model weights. Method and renderer weights are the authors' work, cited above.