farm_uf850_patch_policy

FARM UF850 Patch Policy — task_2 "stack the three cubes" (L1 head)

Reimplementation of Patch Policy (arXiv 2607.18236) as a single-task specialist for the FARM UF850 arm: frozen DINOv2-S/14 dense patch tokens (all 256/camera, base + wrist, no pooling) -> 8-layer / d=512 transformer (26.8M trainable) -> L1-regressed 50-step action chunk. No language conditioning (single task). Trained on 84 episodes, 10 held out.

Training cost: ~0.5 GPU-hours (30k steps, batch 64, one H200, ~24 it/s — episode subset fully RAM-preloaded).

Results (offline, held-out episodes — triage proxy ONLY, no arm rollouts yet)

Open-loop prediction accuracy is a screen, not a verdict — closed-loop rollouts on the arm are the only real test, and this model has not had them.

Inference

from infer_patch_policy import PatchPolicyRunner
r = PatchPolicyRunner("patch_policy_task2_l1.pt")          # downloads DINOv2-S via torch.hub
chunk = r.predict(base_rgb, wrist_rgb, state7)             # (50, 7)

• Inputs: two RGB frames (any resolution; resized to 224x224) + 7-d proprio (6 joint positions rad + gripper). State is normalized internally (stats stored in ckpt).
• Output: 50-step chunk in the FARM dataset action space — 6 relative joint-position deltas (rad) + 1 gripper position [0,1], recorded at 30 fps. Execute like the dataset semantics (apply deltas to current joint positions; gripper is absolute). Re-plan by re-querying at chunk end or earlier.
• Deps: torch >= 2.x, opencv, numpy, internet on first run (torch.hub DINOv2 weights, ~84MB).

Files

• patch_policy_task2_l1.pt — EMA weights (incl. frozen ViT), config, normalization stats.
• train_patch_policy.py — full self-contained training script (data loader for lerobot-v2 FARM episodes, both heads, eval). Repro: see docstring; ~31 min on one H200.
• infer_patch_policy.py — the runner above.

Caveats

Single-task specialist (no language, no multi-task); T=1 observation (no temporal context); diffusion variant underperforms here (simplified MLP denoiser, not the paper's DP head); offline-validated only.