pi05_so101_checkpoint

SO-101 Ball-in-Cup Pi0.5 Policy

A fine-tuned Pi0.5 (π₀.₅) Vision-Language-Action model for the ball-in-cup task using the SO-101 robot arm.

Task Description

Goal: Pick up an orange ball from the table and place it into a pink cup.

Robot: SO-101 - 6-DOF robot arm with gripper

Cameras: Dual camera setup (overhead + wrist-mounted)

Model Architecture

Pi0.5 is a Vision-Language-Action (VLA) model from Physical Intelligence:

The model takes natural language instructions + camera images → outputs continuous joint actions.

Training Details

Inference Performance

JPEG Compression Optimization

We implemented JPEG compression to reduce network transfer time for remote inference:

Architecture

[RunPod GPU Server]              [Robot Mac]
┌─────────────────┐              ┌──────────────┐
│ Pi0.5 Model     │◄── WSS ────►│ run_pi05.py  │
│ (RTX 4090)      │   JPEG      │ (Robot ctrl) │
└─────────────────┘              └──────────────┘

Demo

With JPEG Compression (~270ms latency)

Side-by-side: Overhead camera (left) + Wrist camera (right) - Smooth 3.7 Hz control

Without JPEG Compression (~600ms latency)

Side-by-side: Same task but with raw image transfer - 1.7 Hz control

Sample Evaluation

JPEG Compression (Fast)

5-frame composite: Start → Approach → Grasp → Transport → Final

Raw Images (Slow)

Same task without JPEG optimization

Usage

Server Setup (RunPod)

# Clone OpenPi fork with JPEG support
git clone https://github.com/abdulrahman004/openpi.git
cd openpi
uv sync

# Download checkpoint
uv run huggingface-cli download abdul004/pi05_so101_checkpoint \
    --include "4999/**" \
    --local-dir checkpoints/pi05_so101

# Start server
uv run scripts/serve_policy.py --port 8000 \
    policy:checkpoint \
    --policy.config=pi05_so101 \
    --policy.dir=checkpoints/pi05_so101/4999

Client (Robot Mac)

pip install openpi-client

# Run inference with JPEG compression
python run_pi05.py --server wss://YOUR-POD-8000.proxy.runpod.net

# Or without compression (slower)
python run_pi05.py --server wss://YOUR-POD-8000.proxy.runpod.net --no-jpeg

External Videos (Phone Capture)

Real-world demonstrations recorded externally during evaluation runs:

JPEG Compression (~270ms)

External phone recording showing smooth robot control with JPEG compression

Raw Images (~600ms)

Same task without compression - noticeably slower/choppier control

Edge Retrieval (Out-of-Distribution)

Ball placed at workspace edge - a position that appeared in <10% of training episodes

Comparison with ACT Policy

Trained on the same dataset:

Edge Retrieval: Pi0.5 vs ACT

ACT failed at edge positions - the policy was only trained with ~72 episodes where the ball was mostly in the center/reachable area. When the ball was placed at the edge of the workspace, ACT would miss or fail to reach it entirely.

Pi0.5 succeeds at edge positions despite having the same training data. This demonstrates the power of VLA pre-training:

1. SigLIP (vision encoder) was pre-trained on billions of images - understands "ball" and "edge" concepts generally
2. Gemma (language model) provides semantic grounding - "pick up ball" applies regardless of position
3. Action Expert learned smooth motion primitives from diverse robot arms during base model training

The base Pi0.5 model was trained on data from many different robot arms performing various tasks. This gives it a strong prior on reachable workspace and arm kinematics that ACT (trained from scratch) simply doesn't have.

Infrastructure Notes

Remote Inference Setup:

• Server: RunPod RTX 4090 24GB (~$0.40/hr on-demand)
• Client: Mac Mini M4 controlling SO-101 robot
• Protocol: WebSocket with msgpack serialization
• Optimization: JPEG compression reduces 1.8MB → 71KB per inference

Known Issues:

• RTX 4090 is borderline for memory - occasional OOM during model loading
• US datacenters preferred (2x faster than EU for network transfer)
• First inference takes 30-60s (JAX JIT compilation)

Limitations

• Requires GPU server for inference (not yet optimized for edge deployment)
• Sensitive to lighting changes
• 72 training episodes may limit extreme edge case handling

Citation

@misc{so101_pi05_ball_in_cup,
  author = {Abdul},
  title = {SO-101 Ball-in-Cup Pi0.5 Fine-tuning},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/abdul004/pi05_so101_checkpoint}
}

Acknowledgments

• Physical Intelligence for Pi0.5 and OpenPi
• LeRobot by Hugging Face
• SO-101 robot design community