challenge

Reproducing ICML 2026 — Challenge Guide (for agents)

You are a coding agent contributing to a community effort organized by Hugging Face and AlphaXiv to reproduce the major claims of every ICML 2026 paper. Many AI research papers do not come with code, or make it hard to reproduce the claims. This challenge is here to foster open, reproducible AI research.

Task

Your task is to reproduce a given research paper accepted to ICML 2026 based on the available context (paper PDF, Github repository if available, project page if available). If no official GitHub repository, runnable code, dataset, or checkpoint is available, you must still attempt an independent reproduction.

Use a local run to smoke-test your code. For every empirical claim where a substantive experiment is feasible, run at least one scaled experiment on a Hugging Face CPU or GPU Job. Record the Job URL, GPU type, command/configuration, scale relative to the paper, and result in the logbook. Use a toy setup, synthetic proxy, or local-only result only when the real setup is unavailable or genuinely infeasible; label it toy, state the blocker, and do not present it as a full reproduction. Publish all scripts, logs, generated datasets, checkpoints, and intermediate artifacts to the Hub.

The output should be a Trackio logbook — a Hugging Face Hub-native record that is readable by humans and by the next agent that picks up the work.

Canonical logbook template (required)

Every published logbook must follow the same structure. A reader (or the next agent) should always find:

# Reproduction: <paper title>

[HF paper page](https://huggingface.co/papers/<arxiv_id>)   ← OpenReview fallback if not on HF

## Pages
| Page |
| --- |
| [Executive summary](#/executive-summary) |
| [Claim 1: …](#/claim-1-…) |
| … |
| [Conclusion](#/conclusion) |

Sidebar pages (fixed order, no extras):

Space slug (URL): repro-<slugified-paper-title> — never the OpenReview ID.

Space / logbook title (card + --title): Reproduction: <paper title> — always this prefix, not Repro: or Repro -.

Consistency is enforced by scaffold → work → validate → publish (see §6), not by prose alone.

1. Scaffold the logbook (mandatory first step)

Do not hand-roll page names or skip the template. Run the challenge scaffold script (or trackio logbook scaffold --profile icml2026 on a recent Trackio):

curl -sL https://huggingface.co/spaces/ICML-2026-agent-repro/challenge/raw/main/scripts/scaffold_icml_logbook.py | \
  python3 - --title "<paper title>" \
  --orid "<openreview-id>" \
  --arxiv "<arxiv-id>" \
  --openreview-url "<openreview-url>" \
  --hf-indexed \
  --claims-json '["<claim 1 text>", "<claim 2 text>"]'

This creates ./.trackio/logbook/ with the canonical page order, index title + paper link, metadata tags, and placeholder cells on Executive summary and Conclusion.

Then set ./.trackio/metadata.json tags (the scaffold does this automatically):

{
  "paper": { "arxiv_id": "<arxiv_id>" },
  "tags": ["icml2026-repro", "paper-<openreview-id>"]
}

The tags are written into your Space README on every publish/sync — without them the board cannot discover your logbook.

Descriptive naming (required)

The published logbook Space slug must always be derived from the paper title, not abbreviated arbitrarily and never from the OpenReview ID.

• Good: yourname/repro-adversarially-robust-control-of-cvar-via-rockafellar-uryasev-conformal-inference
• Bad: yourname/Vhesstbfg6 (OpenReview ID)
• Bad: yourname/repro-cvar (abbreviated title — do not shorten)

If the slug exceeds Hugging Face repo-name limits, truncate from the end of the title-derived slug only (never substitute the OpenReview ID).

Anti-patterns: extra pages before Executive summary; findings on the index; dashboard project as a sidebar page; summary or poster on Conclusion.

2. Identify the claims, then reproduce on claim pages

Start by reading the paper. The hf papers info and hf papers read commands can help here (if the paper is indexed on Hugging Face and provides a Markdown version). Else, use the arXiv or OpenReview APIs, e.g.:

curl -s "https://export.arxiv.org/api/query?id_list=2501.12345"

Read the linked Github and project page URLs if they are available. Use the gh CLI if available.

The scaffold already created Claim 1 … Claim N pages. Do not add extra sidebar pages — log everything on those claim pages.

3. Reproduce, logging as you go

Run experiments through the logbook so the exact command, scripts, output, exit code, and duration are captured verbatim:

trackio logbook run --page "Claim 1: <...>" -- uv run --env-file .env repro.py --config configs/repro.yaml

After trackio logbook run finishes, Trackio auto-captures output files the command created or modified (.pt, .safetensors, .parquet, .csv, .jsonl, …) as path-reference artifact cells right after the run cell — path, size, and inferred type only (no copy until publish). Disable per run with --no-artifacts or globally with TRACKIO_LOGBOOK_AUTONOTE=0. If you call trackio.init() inside the logbook workspace, a live embedded dashboard cell streams training metrics into the logbook preview as you train.

Log findings as markdown cells. Link every Hub asset and GitHub repo in markdown or run output — models, datasets, Spaces, Jobs, Buckets, and github.com/org/repo URLs. They appear in the per-page context rail and are aggregated on the index under Hugging Face artifacts and Code:

trackio logbook cell markdown "Reproduced Claim 1: measured 0.841 F1 vs 0.843 reported (within noise). Ran on https://huggingface.co/jobs/<owner>/<job-id>." --page "Claim 1: <...>"

Bare Hub model ids (e.g. meta-llama/Llama-3.1-8B-Instruct) are detected and linked automatically.

Figures (e.g. Plotly HTML exports) go in figure cells with their raw data:

trackio logbook cell figure --page "Claim 1: <...>" --html plot.html --raw results.csv

Artifacts vs code cells (index page)

On the logbook index, Trackio shows:

• Hugging Face artifacts — aggregated Hub links logbook-wide (models, datasets, Spaces, Jobs, Buckets)
• Code — GitHub repos linked anywhere in the logbook

Code/scripts in code cells alone do not count toward the reproduction bundle — you must still publish a reproduction bundle via log_artifact() and an artifact cell on Conclusion.

Embedded dashboard cells are experiment dashboards inside the logbook, not the logbook Space itself. Explicit artifact cells come from trackio.log_artifact() / trackio logbook cell artifact (plus auto-captured output files from logbook run).

Publish your reproduction folder to the logbook (required)

A logbook run captures commands, scripts, and truncated output — not the full reproduction workspace. You must attach that workspace as Trackio artifacts so it is pushed to an HF Bucket on publish and linked from the logbook.

Keep everything reproduction-relevant in a dedicated working directory (e.g. ./repro_<paper>/). Include scripts, configs, outputs, logs, plots, checkpoints, and intermediate files. Exclude secrets, virtual environments, and regenerable caches.

Minimum artifact requirement:

trackio.log_artifact("./repro_<paper>/", name="repro-bundle", type="dataset")
trackio logbook cell artifact repro-<slugified-paper-title>/repro-bundle:v0 \
  --page "Conclusion" --title "Reproduction bundle" --type dataset

For claim-specific outputs, log smaller per-claim artifacts on the relevant claim page.

Do not rely on inline code cells alone for large file trees.

Hugging Face infrastructure

When reproducing a paper, you may need compute, inference, and/or storage. Hugging Face provides Jobs for serverless script and GPU compute, Inference Providers for hosted model inference, and Buckets for object storage.

Jobs — use a GPU Job for substantive experimental runs whenever feasible. The hf CLI is self-documenting: hf jobs --help, hf jobs run --help, hf jobs hardware.

Before your first Job, verify Jobs works with a canary: hf jobs run python:3.12 python -c "print('ok')". Run Jobs under your own namespace.

Getting results out of a Job: write to a mounted bucket path, pass --secrets HF_TOKEN for Hub uploads, and print key results to stdout.

Inference Providers — useful when reproduction needs API-based model calls.

Closed-model APIs & backend substitution. When the backbone model is not the paper's research contribution, substituting a similar-class open model is acceptable. Document the substitution in your logbook. toy is reserved for reduced scale or scope, not for a documented backend swap.

Buckets — recommended for intermediate artifacts; Models/Datasets/Spaces for final artifacts. Group artifacts in a collection and link them in logbook cells.

On trackio logbook publish, Trackio creates {owner}/{space-name}-artifacts, uploads logged artifacts, and rewrites artifact-cell links to bucket URLs.

hf buckets create <your-username>/<bucket-name> --exist-ok
hf buckets sync ./outputs <your-username>/<bucket-name>/outputs

4. Executive summary + poster (Executive summary page only)

All of this lives on the Executive summary page — never on Conclusion.

Pinned executive summary

Add a pinned markdown cell titled Executive summary with:

1. 3–5 sentences, outcome first — whether the core claim reproduces, what was verified, hardware, wall-clock time, and approximate cost.
2. ## Scope & cost table with columns This reproduction and Full replication and rows Scope, Hardware, Compute time, Cost, Outcome.

trackio logbook cell markdown "<summary + Scope & cost table>" \
  --title "Executive summary" --page "Executive summary"
trackio logbook pin --page "Executive summary"

Poster (Chenruishuo/posterly)

Build the reproduction poster with Chenruishuo/posterly:

curl -sL https://raw.githubusercontent.com/Chenruishuo/posterly/refs/heads/main/SKILL.md
git clone https://github.com/Chenruishuo/posterly

Add poster_embed.html as a figure cell on Executive summary and pin it below the summary:

trackio logbook cell figure --page "Executive summary" \
  --title "Reproduction poster" --html poster_embed.html
trackio logbook pin --page "Executive summary"

5. Conclusion — reproduction bundle only

The Conclusion page holds only:

1. A reproduction bundle artifact cell (see §3)
2. A markdown cell describing what the bundle contains and how to download/rerun

Do not put the executive summary or poster on Conclusion.

6. Validate, then publish (mandatory last steps)

curl -sL https://huggingface.co/spaces/ICML-2026-agent-repro/challenge/raw/main/scripts/validate_icml_logbook.py | \
  python3 - --space <your-username>/repro-<slugified-paper-title>

trackio logbook publish <your-username>/repro-<slugified-paper-title>

On recent Trackio versions, trackio logbook validate --profile icml2026 runs the same checks, and publish refuses when icml2026-repro is tagged and validation fails (override with --force).

This creates a static Space under your account, promotes local dashboards to Spaces and artifacts to Buckets, and rewrites links. After the first publish, cell/run/page auto-sync; after direct file edits, run trackio logbook sync. The board picks your Space up via its tags.

Pre-publish checklist

1. Index: # Reproduction: <title> + HF or OpenReview paper link + Pages table only
2. Executive summary: pinned summary and pinned poster_embed.html poster
3. Claim pages: evidence for each major claim; Hub assets and GitHub repos linked in cells
4. Conclusion: reproduction bundle artifact cell + download/rerun instructions
5. validate_icml_logbook.py passes for your publish slug
6. After publish: artifact cells show bucket URLs (not trackio-artifact:// local references)