acrft-yam-extraction

ACRFT — YAM lego-taxi policy-extraction arms

Policy-extraction methods applied to the same pi0.5 base and the same frozen patch critic, as a method-only-diff comparison ring: identical BC init (yam_bc_s300_h30_successonly/100000), critic patch_critic_yam_s347_fixed_tau9_min_200k frozen, and only the extraction objective differs between arms.

Runs suffixed _bb match the BC fine-tune's own training budget: the whole model is trainable (the BC config sets no freeze filter, so matching its budget means matching what it was allowed to move, not only steps and batch), batch 32, constant lr 5e-5, 30k steps. These are whole openpi checkpoints, final step only — load them the way you load any pi0.5 checkpoint.

Runs suffixed _run1 are the earlier pass with the backbone frozen and the action expert only trainable. They are a smaller budget than BC's and are kept for comparison, not as the headline result.

How the weights are delivered follows the suffix, not the arm: a _bb folder is a whole openpi checkpoint (load it directly); a _run1 folder is an action-expert subtree that is overlaid on the BC checkpoint at serving time. CFGRL additionally carries an optimality embedding and is served through its own config.

Each implementation carries file/line-level provenance comments from the official code (or the paper + appendices where no code exists). Two further arms need no weights — QPILOTS-U (test-time critic steering, arXiv 2606.14801) and IDQL/BoN (N-sample argmax of min-ensemble Q, philippe-eecs/IDQL) — they run from the BC checkpoint plus the critic.

Serving

There is ONE serving entry point, scripts/serve_policy.py. Arms reach it two ways, and which way depends on whether the arm changed the policy's weights or only how a chunk is chosen.

Weight-only arms (awr, flowdpg, qam, dql, fqlx) fine-tune the pi0.5 action expert, so they are exported to ordinary openpi checkpoints and served like any checkpoint:

uv run python scripts/export_extraction_checkpoint.py --arm dql          # -> exported/dql_30000
uv run python scripts/serve_policy.py --port 8000 policy:checkpoint     --policy.config pi05_yam_lego_taxi --policy.dir <exported>/dql_30000

CFGRL additionally carries the optimality embedding and samples with classifier-free guidance, which are model properties, so it travels in its own config (with_cfgrl builds the variant of any pi0.5 task config; the guidance weight is cfg_w):

uv run python scripts/serve_policy.py --port 8000 policy:checkpoint     --policy.config pi05_yam_lego_taxi_cfgrl --policy.dir <exported>/cfgrl_30000

Critic-consuming arms need no policy of their own; they are modes of the critic wrapper:

# selection: bon executes the argmax of N draws (this is also IDQL's argmax rule -- label it by N)
uv run python scripts/serve_policy.py --port 8000 --critic <critic_dir> --critic-mode bon --num-samples 8     policy:checkpoint --policy.config pi05_yam_lego_taxi --policy.dir <BC checkpoint>

# implicit: IDQL's implicit policy -- one draw sampled with expectile weights on the advantage
#   --critic-mode implicit --num-samples 64
# qpilots: test-time Q-steering of the sampler, no weights at all
#   --critic-mode qpilots --alpha 0.2
# lps / lpsd / flowdagger: pass their small head
#   --critic-mode lpsd --extraction-head <latent_actor_*.msgpack>
#   --critic-mode flowdagger --extraction-head <flowdagger_run1 dir>

adaptive (execute only the best commitment prefix, then replan) needs a critic trained with several commitment groups, i.e. macro_group_size < horizon; the fixed-chunk critic this ring trains against has a single group, where adaptive is bon under another name.

Caveats

• Offline metrics only so far (critic-Q, held-out demo-MSE, chunk jerk); on-robot success rates are pending. Critic-Q is self-refereed for critic-ascending arms — read it with that in mind.
• Batch sizes differ per arm (4-32) because of VLA-scale memory limits, so at equal step counts the arms have consumed different sample counts.
• Code: 600dba6c581009369f1a927f5117e815d4b1c5ee