gr00t-example_dataset-we3rmfukiq

gr00t model - 🧪 phosphobot training pipeline

• Dataset: base698/example_dataset
• Wandb run id: None

Error Traceback

We faced an issue while training your model.

Traceback (most recent call last):
  File "/root/src/helper.py", line 139, in train_gr00t_on_modal
    trainer.train(
  File "/root/phosphobot/am/gr00t.py", line 1215, in train
    asyncio.run(
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/phosphobot/am/gr00t.py", line 1425, in _call_training_script
    raise RuntimeError(error_msg)
RuntimeError: Training process failed with exit code 1:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/gr00t/data/dataset.py", line 811, in get_data_by_modality
return self.get_video(trajectory_id, key, base_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/gr00t/data/dataset.py", line 679, in get_video
video_timestamp = timestamp[step_indices]
~~~~~~~~~^^^^^^^^^^^^^^
IndexError: index 1090 is out of bounds for axis 0 with size 454

0%|          | 0/29390 [00:02<?, ?it/s]

Training parameters

{
  "validation_dataset_name": null,
  "batch_size": 3,
  "num_epochs": 10,
  "save_steps": 1000,
  "learning_rate": 0.0001,
  "data_dir": "/tmp/outputs/data",
  "validation_data_dir": "/tmp/outputs/validation_data",
  "output_dir": "/tmp/outputs/train"
}

📖 Get Started: docs.phospho.ai

🤖 Get your robot: robots.phospho.ai