TraceGenBenchmark

TraceGen Benchmark Leaderboard

Benchmark: TraceGen Evaluation Suite

We evaluate models on 5 environments using the official TraceGen metrics. Each environment reports MSE, MAE, and Endpoint MSE on held-out test sets.

Test on TraceGen benchmark

Use the official evaluation code provided in: https://github.com/jayLEE0301/TraceGen

Multi-GPU

export CUDA_VISIBLE_DEVICES=0,1,2,3
torchrun --standalone --nproc_per_node=4 \
  test_benchmark.py \
  --config cfg/train.yaml \
  --override \
  train.batch_size=8 \
  train.lr_decoder=1.5e-4 \
  model.decoder.num_layers=6 \
  model.decoder.num_attention_heads=12 \
  model.decoder.latent_dim=768 \
  data.num_workers=4 \
  hardware.mixed_precision=true \
  logging.use_wandb=true \
  logging.log_every=2000 \
  --resume {path_to_pretrained_checkpoint}

Single-GPU

export CUDA_VISIBLE_DEVICES=0
python test_benchmark.py \
  --config cfg/train.yaml \
  --override \
  train.batch_size=8 \
  train.lr_decoder=1.5e-4 \
  model.decoder.num_layers=6 \
  model.decoder.num_attention_heads=12 \
  model.decoder.latent_dim=768 \
  data.num_workers=4 \
  hardware.mixed_precision=true \
  logging.use_wandb=true \
  logging.log_every=2000 \
  --resume {path_to_pretrained_checkpoint}

To reproduce the environment-specific benchmark results reported below, users should evaluate the environment-specific checkpoints TraceGen_{EnvName} from TraceGen Collection, which are trained using data from the corresponding environment only.

Metric definition. All reported errors are computed in a normalized coordinate space: both input images and predicted traces are scaled to the range [0, 1] prior to evaluation. Accordingly, the reported MSE, MAE, and Endpoint MSE reflect absolute errors within the normalized image space.

Submitting to the Leaderboard

• Use the provided evaluation script:
  https://github.com/jayLEE0301/TraceGen
• Report metrics on the official test split, using the corresponding dataset from:
  https://huggingface.co/collections/furonghuang-lab/tracegen
• For environment-specific results, evaluate the corresponding TraceGen_{EnvName} checkpoint.
• Open a PR or submit results via GitHub Issues.