pi0-ego-pipeline

Pi0.5 Egocentric Hand-Pose Extraction Pipeline

Extract hand poses from egocentric video (e.g., Build AI's Egocentric-100K) and convert to LeRobot v3.0 format for π₀.₅ VLA fine-tuning.

Overview

This pipeline:

1. Extracts hand poses using MediaPipe Hands from egocentric video at 3 FPS
2. Computes wrist rotation (rot6D) and gripper aperture from palm landmarks
3. Builds action chunks (50-step future deltas) compatible with π₀.₅
4. Outputs LeRobot v3.0 dataset with observation.state + action columns

Architecture

Egocentric Video (.mp4)
    │
    ▼
┌─────────────────────────────┐
│  HandPoseExtractor          │  ── MediaPipe Hands ──
│  (mediapipe)                │     21 landmarks, wrist pose, rot6D
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  ActionComputer             │  ── Action deltas ──
│  (deltas + chunking)        │     50-step chunks, relative actions
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  LeRobotDatasetBuilder      │  ── LeRobot v3.0 format ──
│  (parquet + mp4)            │     meta/info.json + data/ + videos/
└─────────────────────────────┘
    │
    ▼
LeRobot v3.0 Dataset ──► π₀.₅ fine-tuning

Installation

git clone https://huggingface.co/ShubhamRasal/pi0-ego-pipeline
cd pi0-ego-pipeline
pip install -e .

Note: Uses mediapipe==0.10.18 (legacy solutions API) for headless compatibility. Newer versions require GPU libraries that may not be available in all environments.

Quick Start

1. Process a Single Local Video

python examples/process_local_video.py \
    --video /path/to/egocentric_video.mp4 \
    --output ./output_dataset \
    --task "pick up object" \
    --sample-fps 3.0 \
    --chunk-size 50 \
    --action-dim 32 \
    --state-dim 32

2. Batch Process a Folder of Videos (Build AI style)

python examples/process_buildai_batch.py \
    --video_dir /path/to/egocentric/videos \
    --output ./buildai_dataset \
    --task_map tasks.json \
    --sample-fps 3.0 \
    --chunk-size 50

Where tasks.json maps video filenames to task descriptions:

{
  "factory001_worker001_000000.mp4": "assemble component",
  "factory001_worker001_000001.mp4": "move package"
}

3. Process a HuggingFace Dataset Sample

python scripts/process_hf_sample.py \
    --hf_dataset Voxel51/Egocentric_10K_subset \
    --output_dir ./hf_sample_output \
    --repo_id ShubhamRasal/ego-pi0-sample \
    --num_samples 5 \
    --sample-fps 3.0 \
    --chunk-size 4

4. CLI (single / batch)

# Single video
python -m pi0_ego_pipeline.main single \
    --video my_video.mp4 \
    --output ./dataset \
    --task "perform manipulation"

# Batch from JSON list
python -m pi0_ego_pipeline.main batch \
    --video-list videos.json \
    --output ./dataset

5. Python API

from pi0_ego_pipeline.dataset_builder import EpisodeConfig, LeRobotDatasetBuilder

config = EpisodeConfig(
    video_path="/data/egocentric.mp4",
    output_dir="./dataset",
    task_description="pick up the blue cube",
    sample_fps=3.0,
    chunk_size=50,
    action_dim=32,
    state_dim=32,
)

builder = LeRobotDatasetBuilder(config)
result = builder.process_episode("/data/egocentric.mp4", episode_idx=0)
builder.finalize(tasks=["pick up the blue cube"])

Output Format (LeRobot v3.0)

dataset/
├── meta/
│   ├── info.json              # Schema: state(32), action(32), ego_0_rgb video
│   ├── stats.json             # 1%/99% quantile normalization
│   ├── tasks.parquet          # Task descriptions
│   └── episodes/
│       └── chunk-000/
│           └── file-000.parquet
├── data/
│   └── chunk-000/
│       └── file-000.parquet   # observation.state, action, timestamp, frame_index
└── videos/
    └── observation.images.ego_0_rgb/
        └── chunk-000/
            └── file-000.mp4    # 224×224 @ 3 FPS

Action Representation

Note: chunk_size=50 produces action dimension >32. For π₀.₅, use chunk_size=4 to fit within 32D: 43 + 46 + 4 = 40, or use a custom action head.

Fine-Tuning π₀.₅

After dataset creation, push to HuggingFace and fine-tune:

# Upload dataset to HF
python -c "
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path='./output_dataset',
    repo_id='your-username/your-dataset',
    repo_type='dataset'
)
"

# Fine-tune π₀.₅
lerobot-train \
    --dataset.repo_id=your-username/your-dataset \
    --policy.type=pi05 \
    --policy.pretrained_path=lerobot/pi05_base \
    --policy.dtype=bfloat16 \
    --policy.gradient_checkpointing=true \
    --steps=3000 \
    --batch_size=32

Working with Build AI's Egocentric-100K

builddotai/Egocentric-100K is a gated dataset — you need access approval. Once approved:

from datasets import load_dataset
import os

# Set your HF token
os.environ["HF_TOKEN"] = "your_token_here"

# Load dataset (streaming recommended for 100K hours / 24.7 TB)
ds = load_dataset("builddotai/Egocentric-100K", streaming=True, split="train")

# Iterate and save video files locally
for i, sample in enumerate(ds):
    video_bytes = sample["mp4"]
    metadata = sample["json"]
    
    # Save to local file
    with open(f"video_{i:06d}.mp4", "wb") as f:
        f.write(video_bytes)
    
    print(f"Saved video {i}: {metadata}")

Then process with the batch script:

python examples/process_buildai_batch.py \
    --video_dir ./downloaded_videos \
    --output ./buildai_lerobot_dataset \
    --sample-fps 3.0 \
    --chunk-size 50

Repository Structure

pi0-ego-pipeline/
├── src/pi0_ego_pipeline/
│   ├── __init__.py
│   ├── extractor.py          # MediaPipe Hands → wrist pose
│   ├── action_computer.py    # Action deltas + chunking
│   ├── dataset_builder.py    # LeRobot v3.0 output
│   └── main.py               # CLI entrypoint
├── scripts/
│   └── process_hf_sample.py  # HF dataset sample processor
├── examples/
│   ├── process_local_video.py      # Single video example
│   └── process_buildai_batch.py    # Batch processing example
├── test_pipeline.py          # Unit tests
├── requirements.txt
├── setup.py
└── README.md

Limitations & Future Work

1. No metric depth: MediaPipe outputs normalized z-depth. For metric actions, add depth estimation (MiDaS, ZoeDepth).
2. No camera ego-motion compensation: Build AI data lacks SLAM/camera poses. For moving cameras, add DROID-SLAM or assume static per-clip.
3. No language labels: Use VLM captioning (Qwen2-VL, NVILA) for automatic task descriptions.
4. Fisheye distortion: Undistort frames using per-worker Kannala-Brandt intrinsics before MediaPipe.
5. MANO fitting: For higher-quality hand poses, fit MANO model to 2D landmarks instead of direct palm-plane rotation.
6. Action dimension: chunk_size=50 → action size=500 > π₀.₅'s 32D. Use chunk_size=4 for direct fine-tuning, or customize action head.

How It Works (Research-Backed)

This pipeline follows the approach from recent egocentric-to-robot papers:

• EgoVLA (arXiv:2507.12440): Unified MANO action space, wrist translation + rot6D rotation, 3 FPS sampling
• EgoMimic (arXiv:2410.24221): Camera-centered frame, action normalization, joint human-robot co-training
• VITRA (arXiv:2510.21571): HaWoR hand reconstruction, atomic action segmentation, GPT-4.1 instruction labeling
• π₀.₅ (arXiv:2504.16054): Flow-matching action expert, 50-step chunks, 1%/99% quantile normalization

References

Citation

If you use this pipeline in your research, please cite the relevant papers:

@article{black2025pi05,
  title={$\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization},
  author={Black, Kevin and Brown, Noah and Darpinian, James and others},
  journal={arXiv preprint arXiv:2504.16054},
  year={2025}
}

@article{egovla2025,
  title={EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos},
  journal={arXiv preprint arXiv:2507.12440},
  year={2025}
}

License

This pipeline code is provided as-is for research purposes. The MediaPipe model is under Google's license. Build AI's Egocentric-100K is under Apache-2.0. π₀.₅ is under Gemma license.