vrbhalaaji-gr00t-PenGrab-bjuh7

gr00t Model - phospho Training Pipeline

Error Traceback

We faced an issue while training your model.

Traceback (most recent call last):
  File "/root/src/helper.py", line 166, in predict
    trainer.train(timeout_seconds=timeout_seconds)
  File "/root/phosphobot/am/gr00t.py", line 1269, in train
    asyncio.run(
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/phosphobot/am/gr00t.py", line 1116, in run_gr00t_training
    raise RuntimeError(error_msg)
RuntimeError: Training process failed with exit code 1:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/huggingface/modules/transformers_modules/eagle2_hg_model/modeling_eagle2_5_vl.py", line 253, in forward
f"warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, "
~~~~~~~~~~~~^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

0%|          | 0/1170 [00:13<?, ?it/s]

Training parameters:

• Dataset: vrbhalaaji/PenGrab
• Wandb run URL: None
• Epochs: 10
• Batch size: 27
• Training steps: None

📖 Get Started: docs.phospho.ai

🤖 Get your robot: robots.phospho.ai