pi05-relative-eef-bagplace-memrtc-full-ft-43k

pi0.5 — relative actions, end-effector space (13-dim: xyz + 6D rotation + gripper + base) — bag-place, image augmentation, MEM visual memory + training-time RTC

Full fine-tune of lerobot/pi05_base on a single task — "pick up the bag on the ground and place it on the table" — with actions expressed relative to the observation state at the start of each chunk, and with photometric image augmentation enabled.

This is step 43K of a 100K-step run. The run had not finished when this was published — it was still training, at step 43K of 100K. This is a mid-run snapshot, and held-out loss here is 101% above the run's minimum so far: on this curve it is the worst checkpoint yet, not the best. It is published as a snapshot of where the run had got to, nothing more. If you want the best weights, take the 4K checkpoint instead.

Held-out loss

5% of episodes are held out (10 of 200; 190 train).

The minimum is at 4K (0.0180). The curve rises after it: more steps make this model worse on held-out data, which is why the minimum rather than the endpoint is the checkpoint to prefer. The table above stops where the run had reached when this was published; it was still training and the curve was still climbing.

Short-horizon visual memory (MEM) and training-time RTC

This round enables two things the other bag-place rounds do not, so its numbers are not on the same scale as theirs.

memory_stride is counted in dataset frames, and MEM pretrains on six observations one second apart. The stock default of 30 encodes one second only at 30 fps; this dataset is 50 fps, so the value above is derived from the dataset rather than defaulted.

The eval_loss above is not comparable to the non-RTC rounds

With rtc_training_max_delay non-zero, each example gets a clean ground-truth action prefix of 0–10 steps, and the flow loss is averaged over the remaining postfix only (_reduce_training_rtc_loss). Evaluation runs the same forward path, so the model is scored while conditioned on part of the answer, over fewer positions. That makes the number smaller for reasons unrelated to how well the policy generalises. Comparing it against a round trained without RTC measures the scoring change as much as the model.

Batch is also 32 effective here against 64 in the other rounds — MEM puts memory_frames x cameras images through SigLIP per sample instead of one set — so at equal steps this run has seen fewer samples.

What this does not establish

Enabling MEM on lerobot/pi05_base, pretrained without it, is the paper's weaker MEM-Posttrain-Only ablation; MEM's headline results come from pretraining the video encoder on a large robot+non-robot video mixture that no public pi05 checkpoint provides. MEM's long-horizon language memory (section III-B) is not implemented. Proprioceptive memory (III-D) is off because it is incompatible with relative actions: it gives observation.state a time axis, which to_relative_actions does not accept.

Augmentation

--dataset.image_transforms.enable=true, at most 3 of the following sampled per example. All are photometric; there is no geometric transform, so the image-to-action geometry the policy learns is untouched.

Action representation

use_relative_actions=true with relative_exclude_joints=["gripper"]. This is a processor transform, not a data rewrite: RelativeActionsProcessorStep subtracts the anchor state at batch time and AbsoluteActionsProcessorStep adds it back on the way out, so the policy emits absolute actions and the conversion is identical in training, evaluation and inference.

action[t+k] -= observation.state[anchor]     for all dims EXCEPT gripper
gripper                                       absolute (it is a command, not a pose)

Action dim 13: eef_x, eef_y, eef_z, eef_xx, eef_xy, eef_xz, eef_yx, eef_yy, eef_yz, gripper, base_x, base_y, base_yaw — 12 of 13 dims relative, gripper absolute.

Normalisation is QUANTILES, computed from the relative offsets over chunks of 50 (meta/relative_action_provenance.json records this), not from the absolute targets. Absolute statistics would put most targets far outside [-1, 1].

The end-effector rotation is the stored continuous 6D form (first two rotation-matrix columns), not axis-angle. On this robot the gripper points down, so |rotvec| sits near pi where the sign flips arbitrarily; a built axis-angle view carries frame-to-frame jumps of 2*pi.

Training data

One language-conditioned task, 200 episodes, 477,058 frames (2.65 h at 50 fps), one u850 arm on a mobile base, three cameras (left / right / wrist).

bag-place is also ~7% of the all-10 mixture, so maskjp/pi05-relative-eef-all10-full-ft-100k has already trained on this task. Initialising from lerobot/pi05_base instead makes this a single-task specialist — the clean counterpart to that generalist.

Recipe

Loading

config.json here has had vision_encoder_lr_multiplier and state_in_prompt removed: both are training-only fields from the fork this was trained in, and stock LeRobot rejects unknown config keys. Inference is unaffected.

from lerobot.policies.pi05.modeling_pi05 import PI05Policy
policy = PI05Policy.from_pretrained("maskjp/pi05-relative-eef-bagplace-memrtc-full-ft-43k")

meta/ ships the dataset metadata for deployments that want it. Its info.json has per-dimension names flattened to a plain list; the training datasets store them grouped as {"motors": [...]}, which stock LeRobot resolves to the single key "motors" and which would silently relativise the gripper.

Related

• maskjp/pi05-relative-eef-bagplace-memrtc-full-ft-4k — the same arm at 4K.
• maskjp/pi05-relative-eef-all10-full-ft-100k — the all-10 generalist, which includes this task. Different training data; not a drop-in replacement.