gap-graph-as-policy

GaP — Graph-as-Policy: four-suite evaluation + perception pre-ranking

Code, experiment scripts, execution logs, and per-cell results for reproducing Graph-as-Policy (GaP) on LIBERO, extending it to all four LIBERO suites (the paper evaluates only the grocery/Object suite), and for our perception pre-ranking contribution.

→ Progress timeline — what was done when, including the corrections. → Trial videos — real renderings of the three failure modes, with per-node attribution.

All numbers below are re-aggregated from the raw cell_result.json files in results/. Nothing is estimated. Trials killed by quota, connectivity, or out-of-memory are discarded and re-run — never scored as failures.

---

Results — baseline vs ours

Paired cells only (both arms have valid data for that task); 3 trials per task per arm.

Perception efficiency — the actual result

Read the two tables together: accuracy is a dead heat (39/105 both arms), at 3–10× fewer vision-language-model calls. Pre-ranking is an efficiency result, not an accuracy trick. Where perception is the binding constraint it becomes an accuracy result too — on a deliberately constrained backend, one task moves 6.7% → 80.0% (30-trial ablation, Fisher p ≪ 0.001).

Honest caveats

• Long is unfinished (5/10 tasks paired); that row will change.
• Spatial is a regression for our module (10.0% → 3.3%). Cause is diagnosed, not hidden: graph generation drops the spatial qualifier ("the bowl between the plate and the ramekin" → object_description: "black bowl"), and confident-skip then locks onto the salient bowl rather than the referred one. Fix implemented and unit-tested in v3_perception/, not yet benchmarked.
• 3 trials/task, 1 graph sample/task on a free API key with a daily quota. Sanity-scale, not a full-scale eval.

---

Where GaP works, and why it stops elsewhere

The traces separate three distinct causes — only one is a real capability ceiling.

① A real limit: the skill library. Opening a drawer, turning a stove knob, pushing a plate are not pick-and-place. Those tasks ran their full trials with zero perception aborts and the longest executions measured — 2680–3297 control steps against ~720 for a success (one cell reached 18965). The graph was built, perception verified, the arm genuinely tried; the manipulation primitives do not exist. Visible precisely because the gates passed. E.g. videos/goal_t3.mp4: 2320 steps of genuine attempts, checkpoints failing in order grasp_pose_above_table → bowl_held → drawer_open_detected.

② A fixable pipeline bug: the referring expression is dropped. See Spatial above. E.g. videos/spatial_t0.mp4: two identical black bowls, the graph carrying only "black bowl", and the run failing at target_held — the dominant physical failure outside the grocery suite (0 occurrences in Object, 56 in Spatial).

③ Graph-generation reliability. Two classes of structurally-valid-but-runtime-fatal graphs were found and fixed (see below).

Long is the cleanest illustration: task 1 (two objects into a basket, pure pick-and-place) scores 3/3 in both arms at 1347 control steps — long-horizon is not the problem. Tasks needing drawers or stoves score 0/3 with long executions. Task 0 cannot be compiled into a valid graph at all.

---

Validator fixes (this work)

GaP's validator has 19 structural rules but let two guaranteed-runtime-crash patterns through. Both now reject at codegen time and trigger a resample instead of silently scoring 0/3.

Both are opt-in (strict_exit_coverage=True), enabled only on the codegen path: the executor raises on error-severity issues, so enforcing them there would reject already-generated graphs mid-experiment. Patches + unit tests in patches/; archived rejections in logs/w9_hits.log (10 rejections of the same malformed graph — a systematic model tendency, not a fluke).

A full re-scan confirmed only one completed cell was ever affected by W10 and none of the published Object/Spatial/Goal numbers are contaminated.

---

Repository layout

Models

Free reproduction path: gemini-3.1-flash-lite-preview for both perception and codegen via Google AI Studio's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai). gemini-3.1-pro-preview is 429-rate-limited on the free tier; flash-lite for both roles still reproduces ~0.95 on the grocery suite. Reference: arXiv:2607.05369.

Reproducing

See friend_run/README.md — one API key, one command, resumable. Expected cost is a few dollars of flash-lite calls; ~12 hours on one 24 GB GPU.

API keys are redacted throughout this repository.