test

Soccer Action Detection

Three-stage action spotting for soccer video, built on the pretrained weights in model weights/.

LEVEL 1 — SOCCER KNOWLEDGE      InternVideo2-Soccer
                                players, ball interaction, possession, 15 actions
                                        |
LEVEL 2 — WHEN DID IT HAPPEN?   T-DEED temporal encoder-decoder
                                dense per-frame logits, high-recall candidates
                                        |
LEVEL 3 — SHOULD I TRUST IT?    Verifier: temporal transformer -> XGBoost gate
                                ball/player physics, tracking confidence,
                                hard false positives, frame refinement

Ships with two production classes (pass, pass_received) and scales to all fifteen by editing one list.

---

Read this first: the 25 fps timebase

The annotations are indexed at 25 fps even when the video is not. The sample 5.mp4 is 30.0008 fps / 77,391 frames (43.0 min), and its labels run to frame 64,246 — which is 42.8 min at 25 fps, but only 35.7 min at 30 fps. Read as 25 fps, the goal at frame 23,562 lands on a goalmouth scramble; read as 30 fps it lands on midfield play.

Everything in this codebase therefore works in canonical frames — integer indices on a 25 fps grid. soccer_ad/utils/video.py is the only place that converts, and label_fps is stored per match, so a dataset can mix conventions. Scanning warns loudly if a file's labels cannot fit the video at any candidate frame rate.

---

Quick start

pip install -r requirements.txt

python run_ui.py                 # dashboard on http://127.0.0.1:8000

Then, in the dashboard:

1. Prepare data: import a folder containing videos and their annotations, choose the cache directory, run tracking, and generate soccer-state labels.
2. Model jobs: create a named training job. In its first step, select prepared matches and assign whole matches to training, validation, and test. Choose the output actions here. Each stage tab contains its own model-source field: encoder adaptation accepts the configured InternVideo2 model, an earlier job's adapted encoder, or a custom checkpoint; spotting and OOF independently select their T-DEED temporal initialization or random initialization; the verifier clearly shows that it starts fresh from this job's OOF candidates. Configure encoder adaptation, spotting, OOF generation, and verifier training using the four stage tabs. Each has its own epochs, batch size, learning rates, video settings, freeze schedule, and other advanced settings. Set the OOF fold count and optionally enable automatic advancement. Saving setup validates all four configurations before any training starts. At least two training matches, one validation match, and one test match are required. Keep venue/camera families in the same split.
3. Complete the job's steps in order: encoder adaptation, action spotting, OOF candidate generation, and verifier training. Only the current step's form is shown during training. Each step restores its configuration saved at setup. Start the encoder manually; with automatic advancement enabled, the server starts each following stage on success, even after closing the browser. Automatic steps use their saved configurations, including their own clip and batch sizes; edits to one stage do not overwrite another stage's setup. Progress, metrics, GPU usage, pause/resume, and stop controls remain available.
4. Calibrate & infer appears after all four training steps finish successfully. The job's spotting checkpoint, verifier, and candidate file are selected automatically. Calibration uses its validation matches; test matches remain held out. Select a registered match or browse for a new video to run inference. Click an action in the results to inspect its video frame.

Jobs persist under runs/workflows/<job-id>/. Each records its configuration, dataset metadata, fixed splits, selected actions, per-stage model sources, automation settings, stage status, and artifact references. Stage runs, including their checkpoints, logs, and OOF fold models, live in runs/workflows/<job-id>/runs/<run-id>/. The adapted encoder is exported into runs/workflows/<job-id>/checkpoints/ and passed to later stages automatically. Changing the preparation data/cache folder does not change an existing job's dataset. Failed or interrupted stages can be retried; later stages remain locked. The Activity button shows background work.

Model jobs opens a separate job selection page. Its Create job form is always visible. Create a named job or select an existing job to open its own initialization or training workspace. Use Back to all jobs to return to the selection page. Unsaved setup edits are retained while switching jobs in the same browser session. GPU work queues on the server. Pause job pauses its current training or candidate-generation stage at a safe boundary and prevents automatic advancement. Resume job continues it. Both controls are available in the job summary and directly above the training progress bars once a job starts. A live pause retains its worker. After a server restart or an unexpected failure, Resume stage restores the newest last.pt or paused.pt checkpoint in the same run directory. Recovery continues at the next unfinished batch inside an interrupted epoch; a completed epoch continues at the following epoch. Failures and cancellation never start the next stage automatically.

OOF generation produces held-out predictions for training matches and separate spotting predictions for the job's validation matches. Verifier training respects those explicit splits; the test split is excluded from training and calibration.

Every completed epoch is retained in its run directory as epoch_0001.pt, epoch_0002.pt, and so on. last.pt points to the newest complete training state and best.pt contains the best validation model. Recovery checkpoints include the optimizer, scheduler step, AMP scaler, EMA weights when enabled, configuration, actions, and run state. A paused job can also be stopped cleanly. Training input is checked before GPU allocation, so missing videos, annotations, tracking, and states are reported together. CUDA jobs release unused tensors between stages. With train.auto_recover_oom: true, an out-of-memory failure automatically halves the physical batch, raises gradient accumulation to retain the effective batch, and resumes from the latest completed epoch. Recovery stops at batch size 1 after enabling encoder gradient checkpointing; the job then shows an actionable error and remains resumable.

OOF and ordinary inference use a separate no-gradient micro-batch controller. It begins at train.batch_size, measures real peak VRAM after successful forwards, grows toward train.inference_batch_max_size while retaining the configured reserve, and retries only the pending clips at the last proven or a smaller batch after an ordinary allocation failure. A learned batch is reused only for the same model/input shape and GPU. The same clips and model outputs are used, so this does not change training or inference semantics. CUDA allocator/context asserts still require a process restart. Training itself never silently exceeds its effective batch: set a deliberately larger batch_size * grad_accum_steps when you want the physical training-batch tuner to have more room to grow.

Everything is also available from the CLI:

python cli.py scan                             # discover matches
python cli.py splits --auto                    # write configs/splits.yaml
python cli.py track --split train              # tracking preprocessing
python cli.py states                           # soccer-state pseudo-labels
python cli.py train encoder                    # stage 1
python cli.py train spotting                   # stage 2
python cli.py oof --k 4                        # out-of-fold candidates
python cli.py train verifier                   # stage 3
python cli.py calibrate --apply                # thresholds vs weighted score
python cli.py infer path/to/video.mp4 --run spotting-cli
python cli.py evaluate --split test            # locked-test evaluation

Headless preprocessing stage

The ball/player tracker and soccer-state pseudo-label extraction can run as a self-contained first stage, without starting the dashboard or configuring any model-training settings. Its one JSON file contains every hyperparameter used by this stage, selected match IDs, cache behaviour, and manifest destination.

# Start from the complete, editable template.
python preprocess.py --init-config configs/tracking_states.json

# Edit that JSON, then run tracking followed by state extraction.
python preprocess.py --config configs/tracking_states.json

configs/tracking_states.example.json is a ready-to-copy production starting point. matches: null selects all videos below paths.data_root; use "matches": ["season_1/match_001"] to run a subset. Results are stored as tracking/<match>.npz, states/<match>.npz, preprocessing/features/<match>.npz, and a JSON manifest under the cache root. Each feature archive bundles ball tracks, sparse player tracks, state labels, confidence masks, and metadata for one video. It does not conflict with cache/features, which is reserved for later visual-encoder embeddings.

skip_existing: true (the default) skips a video without opening or decoding it when all its requested outputs already exist. Set it to false to rerun every selected video, or use the individual overwrite_* fields to rebuild a specific cache type. Rebuilding tracking automatically rebuilds its states and feature archive.

To inspect the result, set visualization.enabled to true and select one or more matches. The stage writes an MP4 overlay to data/cache/preprocessing/previews/<match>.mp4; it draws player boxes and track/team IDs, ball trajectory (orange means interpolated), and the current state pseudo-labels. By default it exports the first 1,500 canonical frames (one minute at 25 FPS). This preview reads cached outputs and does not rerun detection for a skipped video.

The template also includes tracking.person_onnx, pointing to the supplied model weights/PlayerDetect/person-detection-model.onnx. The active tracking.detector: "rfdetr" is intentionally the default: it detects both the ball and people in one model, which this state-extraction stage needs. onnx_person is not currently a selectable standalone detector because it does not provide the ball detections required to derive possession and ball state labels.

Any config value can be overridden inline, which makes a sweep a shell loop:

for n in 50 100 150; do
  python cli.py --set data.clip_len=$n --set name=clip_length_$n train spotting
done

Every clip uses all consecutive canonical frames during encoder adaptation, spotting training, validation, and inference. At 25 fps, clip_len: 100 means 100 input frames spanning four seconds, including both the first and last frame. There is no separate sampled-frame count or within-clip sampling mode. Selecting training clip locations still varies across epochs; frames inside each selected clip remain consecutive. Converting a video's native rate (for example 30 fps) to the annotation-aligned 25 fps timeline is a separate step.

model.encoder.temporal_chunk: 8 processes a 100-frame clip in twelve chunks of eight and one chunk of four, preserving every frame. Larger clips increase GPU memory and compute; batch size, gradient accumulation, and gradient checkpointing remain available controls. Old configs containing sampling settings load with a migration warning and use all frames, so re-evaluate older checkpoints under the new input policy. Optional feature caches must have stride 1 and one feature row per canonical video frame.

---

Data layout

A match is a video with a JSON of the same stem beside it. Sub-folders are searched recursively and the relative path becomes the match id.

data/raw/
    2024-09-01_teamA_vs_teamB.mp4
    2024-09-01_teamA_vs_teamB.json
    batch2/
        5.mp4
        5.json                     -> match id "batch2/5"

Annotation format (never modified by the pipeline):

{ "5.mp4": [ {"frame": 111, "action": "pass"},
             {"frame": 139, "action": "pass_received"} ] }

A bare list is also accepted, as are position (ms) and seconds instead of frame. Unknown action names are reported and skipped, not guessed.

---

The 15 classes

This table lives in soccer_ad/constants.py and is the single source of truth: head widths, target shapes, NMS windows, evaluation tolerances and the weighted score all derive from it. Adding a 16th action means adding one row here.

The active classes only control the production head. Stage 1 always trains on all fifteen, and stage 2 keeps a 15-class auxiliary head, because telling a pass from a clearance is exactly the discrimination the detector needs.

---

Headless InternVideo2 + T-DEED training

After tracking/state preprocessing, the first two trainable stages can run as a single UI-free pipeline. It adapts InternVideo2 using raw video, annotations, and state pseudo-labels, exports that soccer encoder, then trains the T-DEED spotter from it. OOF generation and the verifier are not run.

python train_internvideo_tdeed.py --init-config configs/internvideo_tdeed.json
# Edit dataset IDs, batch sizes, model settings, and Hugging Face repo_id.
# Put a write token in HF_TOKEN; do not store the token in JSON.
python train_internvideo_tdeed.py --config configs/internvideo_tdeed.json

The complete template exposes every shared model, data, loss, optimizer, schedule, AMP, loader, and memory-recovery setting, with per-stage recursive overrides. The all-frame policy is enforced: every canonical frame appears in a training clip each epoch, every frame inside each clip is encoded, validation is exhaustive, and cached visual embeddings cannot replace raw video.

Local checkpoints are written after every completed epoch under runs/<run_name>/encoder/ and runs/<run_name>/spotting/; each contains last.pt, metric-selected best.pt, and retained epoch_NNNN.pt files. At every configured five-epoch boundary, current best.pt and last.pt are uploaded to the configured private Hugging Face model repository. The token is read only from huggingface.token_env (default HF_TOKEN).

The configured physical batch is used for the first attempt. The trainer reads peak/free VRAM after each epoch and adjusts the next epoch's physical batch while preserving effective batch size through gradient accumulation. Initial or later ordinary CUDA OOMs restart from the newest safe local checkpoint at a smaller compatible batch; batch 1 falls back to InternVideo gradient checkpointing before reporting an unrecoverable memory error.

configs/internvideo_tdeed.example.json is a concise starting configuration. --init-config writes the fully expanded version when every default field should be visible.

---

Pipeline stages

0 · Tracking preprocessing (optional but needed for the verifier)

RF-DETR (SoccerNet-finetuned) detects ball / player / referee / goalkeeper in one pass. Players go through an IoU tracker with velocity extrapolation, OSNet embeddings are averaged per track and clustered into two teams, and the ball trajectory is assembled with a constant-acceleration Kalman filter that gates out physically implausible detections — which is what rejects the ball on the adjacent pitch.

Missing data is never encoded as (0, 0). Every quantity is stored as value + valid mask + confidence, plus interpolated and frames_since_seen.

Runtime. Detection dominates. RF-DETR runs ~1.4 fps in fp32 and ~6.5 fps in fp16 on a 6 GB RTX 2060. A 45-minute match is ~64,500 canonical frames, so at detect_stride=2 that is ~2.4 h/match on that card — roughly 143 h for 60 matches. On a 4090/A100 expect several times faster. detect_stride divides the cost directly; above ~3 the ball trajectory starts losing fidelity while player positions tolerate much more. Stages 1 and 2 do not need tracking.

1 · Encoder soccer adaptation

InternVideo2-B (768-dim, 12 blocks, patch 14, tubelet 1) plus five heads: a 15-class action head and four soccer-state heads (possession, ball state, player–ball relation, possession transition) trained on tracking-derived pseudo-labels.

L = L_action + 0.25·L_possession + 0.20·L_ball_state
             + 0.20·L_relation   + 0.30·L_transition

Every state term is confidence-weighted and masks its unknown slot, so a match with poor tracking still contributes its action supervision in full while contributing nothing false about possession.

Three phases run as one freeze schedule: head warm-up → upper half → everything with layer-wise LR decay. Output: checkpoints/internvideo2_soccer_best.pt, the reusable encoder every later stage starts from.

2 · Spotting

RGB clip -> InternVideo2-Soccer -> temporal adapter -> T-DEED -> dense logits

The encoder is pooled spatially only — the temporal axis survives to the heads. With the default tubelet size of 1, the adapter refines all 100 frame features and projects their width from 768 to T-DEED's 368. The temporal length stays at 100 throughout; there is no interpolation from sparsely sampled frames.

Each ground truth becomes two targets:

• broad — a linear ramp, 1 at the event, 0 at that class's tolerance. Buys recall.
• exact — a narrow Gaussian (σ ≈ 2–3 frames). Buys frame-exact localisation.

plus a displacement regression that snaps a confident-but-offset peak onto the event. Per-class sigmoids, not a softmax: actions genuinely co-occur, and each class needs its own calibrated threshold.

The published 15-action weights are applied whenever loss.use_class_balance is enabled (the default). Stage 1 applies them to all 15 action outputs. Stage 2 applies them to the active production classes and independently to its 15-class auxiliary head. Thus the current pass / pass_received head uses a 1.0:1.4 ratio, while goal already receives a 10.9:1 ratio relative to pass in the auxiliary objective and will keep that ratio when it is added to the production class list. Stage 3 also weights each candidate's classification and frame- refinement losses by its claimed action; its final XGBoost models remain separate per action.

Validation reports the weighted competition score on discrete events, not a loss — a loss can improve while detection gets worse.

3 · Verifier

candidate C -> C±30 -> multimodal tokens [61, D]
                          |
              3-layer / 192-D / 4-head temporal transformer
                          |
        +-----------------+------------------+
        v                                    v
  192-D embedding                     61-frame heatmap -> refined frame
        |
  [embedding || T-DEED stats | ball stats | player stats
              | possession stats | tracking-quality stats]
        |
     XGBOOST  ->  P(valid)  ->  class threshold  ->  keep / reject

The transformer reads the shape of the window; the boosted tree makes the accept/reject call from that embedding plus ~48 engineered scalars, which is where trees beat a small network on limited data. Physics is fused into the visual token through a reliability gate driven by tracking quality, so a failed track degrades to the visual representation instead of poisoning it — reinforced by deliberately corrupting tracking on ~50% of training samples.

Trained on out-of-fold candidates: K models, each predicting the fold it never saw. A verifier trained on in-fold candidates learns that stage 1 is trustworthy, which is the opposite of its job.

4 · Calibration

Calibration selects the confidence threshold and NMS window for each action using labeled out-of-fold or validation predictions. The threshold controls which candidates survive; NMS suppresses nearby duplicate detections. The search maximizes the weighted mean of per-class F1 used by this project. Positive class weights change the contribution of each class to that score; they do not by themselves justify accepting more false positives within a class.

In Calibrate & infer, use Browse predictions... to select the candidate JSON. Their match IDs must map to the registered videos and annotations. Review the suggested settings, then apply them to the active configuration if desired. Calibration changes postprocessing parameters and trains no neural weights. Keep test matches untouched until all training and calibration choices are final.

---

Maintenance

Adding matches

python cli.py scan                     # picks up the new files
python cli.py track                    # only new matches are swept; caches are reused
python cli.py states
python cli.py train spotting           # fine-tune on the expanded set
python cli.py oof --fresh              # regenerate candidates if this is a major retrain
python cli.py train verifier
python cli.py calibrate --apply

Nothing needs rebuilding if the tracking models have not changed.

Adding an action

Edit actions: in the config (or tick it in Prepare data). The spotting head grows in place — SoccerSpottingModel.expand_actions copies the existing class rows so a trained pass/pass_received model keeps its pass detectors and only the new rows are initialised. Then fine-tune stage 2, retrain the verifier with that class's hard negatives, and recalibrate.

Replacing the ball/player detector

Re-run tracking → states → verifier. The RGB representation is untouched, so stages 1 and 2 do not need rebuilding unless the new state labels are substantially better.

---

Layout

soccer_ad/
  constants.py          15-class taxonomy, weights, tolerances, canonical fps
  config.py             typed layered config (pydantic) + YAML _base_ inheritance
  utils/video.py        canonical-timebase video access  <- the fps conversion
  data/                 ingest, splits, labels, targets, datasets, augmentation
  tracking/             detector, player tracker, ball Kalman, teams, cache
  states/               soccer-state pseudo-labels with confidences
  models/               internvideo2, tdeed, adapter, heads, spotting, verifier
  training/             engine, losses, optim, per-stage trainers, OOF
  eval/                 tolerance-aware metrics, NMS, calibration
  inference/            end-to-end video -> events
  server/               FastAPI + dashboard (static/, no build step)
configs/                default.yaml + presets/
cli.py                  scriptable equivalent of the dashboard
run_ui.py               dashboard entry point

Presets

---

Pretrained weights

T-DEED's ks=9, k=4 were recovered from the checkpoint: up_size = round((ks+1)·k) forced odd gives the 41-tap convkw kernel it actually contains, and num_layers=2 is what produces its 5 SGP blocks + 2 mixers.

---

Notes on this environment

• PyAV is blocked by an Application Control policy on this machine, so video decoding falls back to OpenCV. The OpenCV path seeks once per clip and then uses grab()/retrieve() to decode the complete canonical clip. If PyAV becomes available, set data.decode_backend: pyav for a further speedup.
• onnxruntime is CPU-only here. Install onnxruntime-gpu if you want the ONNX player detector path to be fast; the default RF-DETR path uses PyTorch and is unaffected.
• The RTX 2060 is Turing, so amp_dtype: fp16 (not bf16) — see the smoke preset.