adungus-ACT_BBOX-PP-1-2

act Model - phospho Training Pipeline

Error Traceback

We faced an issue while training your model.

Training process failed with exit code -6:
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/lerobot/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/lerobot/lib/python3.10/site-packages/torchvision/ops/misc.py", line 62, in forward
return x * scale + bias
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.49 GiB. GPU 0 has a total capacity of 22.06 GiB of which 4.12 GiB is free. Process 1 has 17.91 GiB memory in use. Of the allocated memory 17.50 GiB is allocated by PyTorch, and 109.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
wandb:
wandb: 🚀 View run act at: https://wandb.ai/addverb/phospho-ACT/runs/ru1psx03
wandb: Find logs at: ../data/phospho-app/adungus-ACT_BBOX-PP-1-2/1753939361.404905/wandb/run-20250731_073432-ru1psx03/logs
terminate called without an active exception

Training parameters:

• Dataset: phospho-app/PP-1_bboxes
• Wandb run URL: None
• Epochs: None
• Batch size: 100
• Training steps: 100000

📖 Get Started: docs.phospho.ai

🤖 Get your robot: robots.phospho.ai