WCA-Bench-baselines

WCA-Bench example results

This directory contains baseline results produced on the official WCA Results Export (format v2.0.2, export date 2026-09-21) over a sampled test window.

Everything here is regenerated by the commands in Reproducing below.

Dataset scale (full build)

Evaluation setup

scripts/run_all_baselines.py --mode small samples a handful of competitions inside the test window and takes a representative subsample of the training split, so that the full baseline suite completes on a single machine. The numbers below are therefore indicative, not final — the public leaderboard must be re-run on the full dataset under the rolling-window protocol.

The device is recorded per baseline in every report: pure tabular baselines report device: "cpu", while the LSTM, the GNN and the boosting baselines report device: "cuda:NVIDIA GeForce RTX 4060 Laptop GPU" together with gpu_hours.

Files

Each report uses the same top-level schema:

{
  "task": "...", "model": "...",
  "overall": {...},
  "stratified": {"by_event": {...}, "by_skill_level": {...}, "by_time_slice": {...}, "by_continent": {...}},
  "hard_subset": {...},
  "significance": {"reference": "...", "metric": "...", "paired_unit": "...", "n_pairs": 0,
                   "mean_diff": 0.0, "ci95": [0.0, 0.0], "p_value": 1.0,
                   "effect_size": {"name": "cohens_d", "value": 0.0}, "test": "paired_t", "seed": 42},
  "cost": {"mode": "small", "baseline_kind": "...", "device": "cuda:...",
           "wall_clock_sec": 0.0, "cpu_hours": 0.0, "gpu_hours": 0.0, "gpu_model": "..."},
  "extras": {...}, "elapsed_sec": 0.0, "seed": 42, "data_summary": {...}
}

Leaderboard summary

Single run (seed 42). See leaderboard.md for the complete tables.

Multi-seed view (see multi_seed.md for the full table):

Two findings worth noting:

1. Domain-aware baselines beat generic deep and graph models. On T1 the LSTM (MAE(log) 0.9436 ± 0.0754) is far behind gradient-boosted trees (0.1055 ± 0.0417); on T2 the GNN (τ 0.7440 ± 0.1258) trails the rule-aware Psych Sheet baseline (0.8132 ± 0.0342).
2. Rank stability depends on the task. On T1–T2 the multi-seed ordering matches the single run. On T3 the single run favours xgboost_dnf (0.3731) while the multi-seed mean favours beta_binomial (0.2938 ± 0.0570 vs 0.2779 ± 0.0724) — the sampling variance of the evaluation window is large enough to reorder close contenders, which is exactly why multi-seed reporting is mandatory.

Reproducing

python -m venv .venv
.venv\Scripts\activate            # Windows
# source .venv/bin/activate       # macOS / Linux
pip install -e ".[dev,fast,boost]"
pip install -e ".[deep]"          # optional: torch for the LSTM / GNN baselines

python scripts/download_data.py                  # fetch the official TSV export
python scripts/build_dataset.py --source raw     # decode, featurize, split, freeze statistics
python scripts/run_all_baselines.py --mode small --device cuda
python scripts/build_leaderboard.py --report-dir outputs/reports --out-dir examples
python scripts/run_multi_seed.py --seeds 42 43 44 --mode small --device cuda \
    --out-dir outputs/multi_seed --examples-dir examples

Use --device cpu (the default) for a CPU-only, fully reproducible run; --device auto selects CUDA when it is available.

Offline / CI path, using synthetic data:

python scripts/generate_synthetic.py --small
python scripts/build_dataset.py --source synthetic
python scripts/run_all_baselines.py --mode small
python scripts/build_leaderboard.py