dinov2-latent-wm-3cam-threading-d0

3-camera latent world model — DINOv2 → 56-d bottleneck (MimicGen threading_d0)

Same 56-dimensional bottleneck as chomeed/dinov2-latent-wm-threading-d0, but reading three camera views instead of two. The third is a purpose-built view of the tripod's insertion bore, which the original two cameras never see.

Why a third camera

agentview is a fixed third-person view; robot0_eye_in_hand looks straight down the gripper axis, so the needle tip occludes the very thing the task is about. Neither camera can see the hole. Object-pose error was consequently dominated by the tripod, especially on failure rollouts where the tripod has been knocked out of position — precisely the state no existing view resolves.

How the camera pose was obtained

Not by hand-tuning angles. Three attempts at guessing pitch/yaw offsets from the wrist mount failed: small tilts changed nothing visible, large ones swung the view onto the arm, and offsetting the mount backwards put the camera inside the Panda hand body. Two non-obvious bugs had to be fixed first, and both are worth knowing if you re-derive this:

1. Camera writes appeared to do nothing. The wrist camera is attached to the robot0_right_hand body. MuJoCo renders from data.cam_xmat, which mj_forward derives from model.cam_quat — so writing the quaternion after sim.forward() leaves every render using the stale matrix. Measured directly: render diff 0.0 without a re-forward(), 77.1 with it. Every candidate angle had been rendering the baseline view.
2. The frame was wrong. Previews sampled a fixed fraction through the episode, where the gripper is still far from the tripod. No camera angle can show a hole the needle has not reached.

The working method solves for the pose instead of parameterising it ([cam_lookat.py]):

• pick the frame of closest needle–tripod approach automatically (argmin ‖needle − tripod‖);
• place a viewpoint on a sphere around the needle/tripod midpoint at (azimuth, elevation, radius);
• construct the rotation that looks at that midpoint (MuJoCo convention: camera looks down −Z, +X right, +Y up), so every candidate frames the target by construction;
• convert the world pose into the hand-body-local cam_pos / cam_quat MuJoCo actually stores, yielding a fixed, physically-realisable mount rather than a per-frame gaze.

A 32-view azimuth × elevation contact sheet then chose the pose. Only az 240–300 at low elevation has a clear line of sight — at az 45–135 the Panda hand fills the frame — and of those, az 260 is close enough to the bore axis to see through the hole rather than at its face.

The chosen mount

azimuth 260°, elevation 15°, radius 0.15 m from the needle/tripod midpoint
solved at: mimicgen_threading_d0_224x224, episode 0, frame 188 (closest approach)

frame          robot0_right_hand
cam_pos_local  [-0.198150, -0.035687,  0.078038]      # 21.6 cm from the hand
cam_quat_local [-0.322727,  0.517794,  0.611714, -0.503530]   # w, x, y, z

Reproduce with RAD=0.15 AZ=260 ELEVS=15 python cam_lookat.py. The pose is also shipped in each dataset at meta/insertion_camera.json.

Honest limitations of the mount

• 21.6 cm from the hand is a boom, not a compact wrist camera. This is forced by geometry: the target sits below the hand, so any low-elevation view is far from it. Pulling in under ~18 cm requires elevation ≥ 35°, which flattens the needle against the post and loses the alignment gap.
• The tripod is out of frame for roughly the first half of each episode. The mount is rigid to a moving hand, and early on the hand is nowhere near the target. This is an insertion-phase view — it is meant to be used alongside the other two cameras, never instead of them (the ablation below shows exactly that).

Results

ratio = latent prediction MSE ÷ copy-the-current-latent MSE, so 1 − ratio is the fraction of frame-to-frame latent motion the dynamics explains. Both rows are latent=56, conv, hist=1, horizon=8, n=40000, 25k steps — identical but for the camera count.

Pose error improves on every split — expert −20% (2.83 → 2.27 mm), flow_success −28% (4.23 → 3.04), flow_failure −25% (20.40 → 15.20) — while the dynamics ratio improves 16% (0.214 → 0.179).

Reconstruction gets worse (0.202 → 0.248), which is the expected trade: the same 56 dimensions must now reproduce three views' tokens instead of two. The latent pays for the extra view in reconstruction fidelity and is rewarded in prediction and geometry — the third camera adds structure that is more predictable than what it displaces.

Does the third view actually help? — view ablation

One-factor test: load the 3-camera data once, then train the same frozen-DINOv2 soft-argmax pose readout on view subsets of it. Identical frames, episodes, split assignment, seed, init and step count — the only difference is which cameras the readout may look at. Pose is read at the current frame, so this measures what the observation affords, independent of dynamics.

All values are millimetres; offset = needle − tripod is the decision metric, since relative pose is what a controller consumes.

The cameras divide by object. The wrist camera owns the needle — dropping it costs 1.89 → 2.33 mm (+23%), because it is rigidly attached to the gripper holding the needle and sees it at near-fixed relative pose. The insertion camera owns the tripod — dropping it costs 9.10 → 15.16 mm on failure rollouts (+67%), because it is the only view aimed at the bore. Neither pair suffices: each object has a different best observer, which is why the third view supplements rather than replaces.

Adding it improves every split: expert offset −20%, failure offset −43%. That last number is the consequential one — the displaced-tripod regime that dominated every pooled metric was being estimated by cameras that could not see the displaced object.

Contents

encoder, dynamics, decoder, pose_probe (safetensors) plus config.json carrying every normalisation statistic (prop_mean/std, target_mean/std, token_mean/std) and the measured pred_mse / copy_mse / ratio. Every statistic needed to run the checkpoint is in config.json — normalise tokens with token_* before the encoder, proprioception with prop_* before the dynamics, and de-normalise probe output with target_*.

3 cams (B,3,3,224,224) → frozen DINOv2-small → (B,3,256,384)   # 16x16 grid, prefix=1 (CLS only)
  → Enc (conv, LayerNorm) → z ∈ R^56
  → Dyn: z_{t+1} = z_t + MLP([z_t, action_chunk(8x7), proprio(9)])
RL state = [z (56), proprio (9)] = 65 dims

hist=1 (current frame only) and horizon=8 (one 8x7 action chunk = one decision step). The latent is never supervised on object pose — trained only on token reconstruction, latent dynamics and future proprioception; pose is read out afterwards by a probe, for evaluation only.

Data

chomeed/mimicgen_threading_d0_224x224_insertcam and its _mtdit_flow_55k_{success,failure}_insertcam siblings — the original datasets with observation.images.insertion added by sim-state replay, encoded to match the existing views exactly (AV1, yuv420p, 20 fps, crf 30).

Caveats

• Single seed. Measured seed variance on this rig is σ ≈ 0.15 mm, so the 2.20 → 1.77 mm gap is well outside noise, but the run has not been seed-repeated.
• Reconstruction is worse than the 2-camera model (0.248 vs 0.202) — the same 56 dims must now reproduce three views. The latent pays for the extra view in reconstruction while getting better at prediction.
• target=tokens, not segdepth. The seg/depth/RGB target caches cover only the original two cameras; the segdepth objective (which wins in every matched 2-camera comparison) needs those rendered for the third view first.