pi05-relative-eef-full-ft

pi0.5 — full fine-tune, relative actions, end-effector space (13-dim pose, 6D rotation)

Fine-tuned from lerobot/pi05_base on the 949-episode base4 multi-task mixture (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz), with every parameter trainable — 4,143,404,816 of them: vision tower, language model and action expert. Contrast with the frozen-VLM arm, maskjp/pi05-relative-eef-frozen-vlm, which trains only 693M.

This is step 5000, not the end of the run. The schedule ran to 30000, but held-out loss bottoms early and then climbs:

Train loss keeps falling to ~0.008 throughout. 4.14B parameters memorise 900 episodes in well under half an epoch. 30K is the wrong schedule for full fine-tuning on this dataset; ~5000 steps, or early stopping on held-out loss, is the right one.

Is full fine-tuning worth it here?

No. The frozen-VLM arm is 1.1% better despite training 6x fewer parameters, and it is still improving at 30K rather than overfitting.

config.json differs from training

Training set vision_encoder_lr_multiplier=0.1, putting the vision tower and multi_modal_projector on a 2.5e-6 learning rate against 2.5e-5 for the expert and language model. That field does not exist in stock lerobot 0.6.2, and draccus rejects unknown config keys, so it has been removed from the uploaded config.json — the model loads on unmodified lerobot. The field is read only by get_optim_params and has no effect at inference, so nothing about this model's behaviour changes. train_config.json retains it as the training record.

To reproduce the training, you need lerobot at commit fbb811fc plus that field; inference needs neither.

Camera sensitivity

Not measured for this arm. The matched frozen-VLM models score 0.088-0.091 on the camera-swap sensitivity ratio — far below the 0.5 grounding threshold — meaning they predict arm motion largely without reading their cameras. Whether unfreezing the vision tower changes that is an open question this card does not answer.

Deploying without the training dataset

train_config.json records an absolute dataset.root from the training machine and the l5vel-peng/multitask-eefrel-h50 view is not published, so a robot can resolve neither. Nothing here needs it: normalisation statistics are baked into policy_preprocessor_step_3_normalizer_processor.safetensors, and action_feature_names lives in config.json.

from lerobot.configs.policies import PreTrainedConfig
from lerobot.policies.factory import get_policy_class, make_pre_post_processors

repo = "maskjp/pi05-relative-eef-full-ft"
cfg = PreTrainedConfig.from_pretrained(repo)
cfg.device = "cuda"
policy = get_policy_class(cfg.type).from_pretrained(repo, config=cfg)
pre, post = make_pre_post_processors(cfg, pretrained_path=repo)

Avoid make_policy(cfg, ds_meta=...): a non-None ds_meta makes LeRobot overwrite action_feature_names from the dataset, which is the path that needs the missing files. If your deployment insists on it, this repo also ships meta/ (~1.8 MB) — point dataset.root at a directory containing it, no data/ or videos/ required.

Action representation

Targets are relative: action[t+k] - state[anchor], one anchor per chunk, added back after inference. The gripper stays absolute (relative_exclude_joints=['gripper']).

Configuration

pretrained_path lerobot/pi05_base
chunk_size 50   n_action_steps 10   n_obs_steps 1
freeze_vision_encoder false   train_expert_only false   (all 4.14B trainable)
gradient_checkpointing true   compile_model true   dtype bfloat16
optimizer_lr 2.5e-5, warmup 1000, cosine to 2.5e-6 over 30K
vision tower + projector at 0.1x that (2.5e-6)
norm  VISUAL IDENTITY | STATE QUANTILES | ACTION QUANTILES
batch 21/rank x 3 GPUs = 63 effective, DDP, seed 1000

Apache-2.0, inherited from LeRobot.