mtdit-spatial-relative-joints

MTDiT-Spatial — relative actions, joint space (10-dim motor targets)

mtdit_spatial with vision_feature_mode=cls, which is byte-identical to stock multi_task_dit. Trained from scratch on the 949-episode base4 multi-task mixture (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz).

Checkpoint step 060000 of a 100K-step run — the held-out-loss minimum, not the final step (unique minimum; the curve turns and ends at 0.0084 by 100K).

What this is for

One cell of a 2x2 matrix crossing action representation (relative joints vs relative end-effector) with policy (MTDiT vs pi0.5), testing whether a relative action target makes the policy attend to its cameras. It does not. These weights are published as a negative result.

Measured on croissant held-out episodes 93-97, arm dims 0:6 (joint angles, degrees).

Camera sensitivity holds observation.state fixed, swaps in another episode's cameras from a state-matched frame, and divides the resulting change in predicted arm joints by the natural cross-episode spread of the ground truth. 1.0 means the policy tracks the scene; 0.0 means it ignores it. Measured on croissant held-out episodes 93-97 restricted to the "pick up the croissant" phase, four inference seeds under common random numbers, padded steps excluded, with the null test (each variant fed the anchor's own cameras) required to return 0.

Read the ratio together with the prediction error: a high ratio with a high error is an unstable policy moving because any input moved, not a grounded one. The error here is well inside the 0.8 reliability gate.

Action representation

Targets are relative: action[t+k] - state[anchor], one anchor per chunk, added back after inference. The gripper stays absolute (relative_exclude_joints=['gripper']) because it is a near-binary command with a 70-unit single-step jump, not a pose.

Normalisation statistics must match the representation, so this trains against l5vel-peng/multitask-relative-h32, a metadata-only sibling of the mixture whose meta/stats.json holds offset quantiles computed at the policy horizon. Training against absolute statistics would squash every target.

Configuration

horizon 32   n_action_steps 24   n_obs_steps 2
hidden_dim 512   num_layers 4   num_heads 8   dropout 0.1
diffusion / DDPM, num_train_timesteps 100
optimizer_lr 3e-4   weight_decay 0   cosine to 100K, warmup 0
CLIP openai/clip-vit-base-patch16, vision_encoder_lr_multiplier 0.1
images resize [240,320] -> random crop [224,224]
norm  VISUAL MEAN_STD | STATE MIN_MAX | ACTION QUANTILES
batch 64, bf16, seed 1000

Loading

from lerobot.policies.factory import make_policy   # needs lerobot_policy_mtdit_spatial installed

Requires the lerobot_policy_mtdit_spatial plugin, which registers the mtdit_spatial policy type. Loading with --policy.path inherits this checkpoint's config; --policy.pretrained_path loads weights only and resets stored settings to defaults.

Apache-2.0, inherited from LeRobot.