robopro-gb10-eval-guide

RoboPRO eval on GB10 — setup guides

How to evaluate VLA policies on the RoboPRO / RoboTwin benchmark on an NVIDIA GB10 (Grace-Blackwell) SLURM cluster — including running the model and simulator on separate nodes (for models too big to co-locate with the sim).

Architecture

The eval splits into two processes talking over a TCP socket (4-byte big-endian length header + JSON-with-numpy payload):

• Simulator — SAPIEN + the RoboPRO task, runs in-process in the robotwin env; produces observations (3-cam RGB + proprio + instruction) and executes the returned actions.
• Model — the VLA policy in the xvla env; turns each observation into a 30-step action chunk.

The obs→action flow is fixed (sim has the obs, model returns the action); either side can be the socket server — choose by env flags:

Both work single-node (localhost) or cross-node (set the peer host/bind + a shared port). Use sim-as-server if that's your convention or if your orchestration starts the sim first.

Documents

• EVALUATION_PIPELINE.md — START HERE to recreate the pipeline. The full end-to-end reference: directory layout, node-local staging, the per-node worker, the work-stealing lease queue (+ resume), every env var, copy-paste run recipes, output/scoring, the orchestrator patterns, and a recreate-elsewhere checklist.
• ROBOPRO_EVAL_CROSSNODE_GB10.md — model-as-server, split across nodes (the default direction). 2-node SLURM run steps; the MODEL_SERVER_BIND (server) / MODEL_SERVER_HOST (client) + --port knobs; how to plug in your model.
• ROBOPRO_EVAL_SIM_AS_SERVER.md — sim-as-server alternative (sim binds, model connects). The SIM_AS_SERVER / MODEL_AS_CLIENT flags + SIM_SERVER_HOST/SIM_SERVER_PORT. Both modes are additive/flag-gated and validated end-to-end (a full episode completes with success).
• GB10_EVAL_PIPELINE.md — read this for the gotchas. GB10-specific issues + the validated config: SAPIEN 3.0.0b1 (3.0.3 regresses office/kitchen), WORKERS_PER_GPU=1 (MPS broken on Blackwell), OptiX denoiser (OIDN is a silent no-op), node-local staging, ffmpeg PATH, the exit-144 pkill -f footgun, etc.

Quick start

1. To recreate the whole pipeline, read EVALUATION_PIPELINE.md — it has the layout, the run mechanism, and copy-paste commands. On this GB10 cluster the built setup is group-readable at /shared_work/jack/eval_root (run it in place or copy it).
2. Read GB10_EVAL_PIPELINE.md and set the validated knobs (DXVLA_SAPIEN_B1=1, WORKERS_PER_GPU=1, DXVLA_DENOISER=optix, node-local /tmp staging).
3. Pick a connection pattern (table above) and follow that guide's copy-paste SLURM commands.
4. Smoke-test one task (EVAL_TEST_NUM=1) and confirm a _metrics.jsonl line with "success" appears and both sides log "connected" — then scale up.

Plugging in your own model

The model side loads the policy via get_model() (policy/dxvla/deploy_policy.py). Either swap a checkpoint (CKPT_NAME + code dir; see each ckpt's EVAL_INTERFACE.md), or implement get_model() returning an object with generate_actions(input_ids, image_input, image_mask, domain_id, proprio, ...) -> [B, num_actions, action_dim]. The socket/framing is handled for you; you only provide the policy.