dinov2-latent-wm-265d-threading-d0

265-d latent world model — DINOv2 bottleneck (MimicGen threading_d0)

A wider-bottleneck sibling of chomeed/dinov2-latent-wm-threading-d0 (56-d): identical data, encoder, dynamics, horizon and step count — only the latent width differs. Two cameras (agentview, robot0_eye_in_hand), hist=1, horizon=8 (one 8×7 action chunk = one decision step).

Results

All rows: conv encoder, hist=1, horizon=8, n=40000, 25k steps, 2 cameras. Millimetres, per split.

Widening the bottleneck 4.7× buys almost nothing. On the token objective it is 6% better expert offset (2.83 → 2.67 mm); on the segdepth objective it is 2% (1.96 → 1.92 mm), which is inside the σ ≈ 0.15 mm seed noise measured on this rig. The gain that does exist is entirely in the tripod term (0.88 → 0.69 mm); needle is flat or slightly worse.

The useful conclusion is the negative one: 56 dimensions was already enough. What the latent was missing is not capacity — it is information the two cameras never captured. For comparison, adding a third camera that actually sees the insertion bore reaches 2.27 mm at 56 dims (…-3cam-…), beating 265 dims at two cameras (2.67 mm). A better view is worth more than 4.7× the bottleneck, and costs nothing in state size.

Do not compare ratio across latent widths

ratio = pred_mse / copy_mse is reported per run, but it is not comparable between these models:

LayerNorm pins each latent to norm √L, so with more dimensions the per-dimension change between t and t+8 shrinks and the copy baseline becomes much stronger — copy falls 6.7× from 56 to 265 dims. The ratio therefore looks worse at 265 while absolute prediction error is 4.6× better. Compare ratios only within a fixed latent width (and a fixed horizon); across either, use the millimetre numbers.

Contents

• tokens265/ — decoder reconstructs frozen DINOv2 tokens
• segdepth265/ — decoder reconstructs {segmentation, depth, RGB} at 56×56 (the stronger objective)

Each holds encoder, dynamics, decoder, pose_probe (safetensors) and a config.json carrying every normalisation statistic (prop_mean/std, target_mean/std, token_mean/std) plus the measured pred_mse / copy_mse / ratio. Normalise tokens with token_* before the encoder, proprioception with prop_* before the dynamics, and de-normalise probe output with target_*.

2 cams (B,2,3,224,224) → frozen DINOv2-small → (B,2,256,384)    # 16x16 grid, prefix=1
  → Enc (conv, LayerNorm) → z ∈ R^265
  → Dyn: z_{t+1} = z_t + MLP([z_t, action_chunk(8x7), proprio(9)])
RL state = [z (265), proprio (9)] = 274 dims;  replay buffer 51 MB at 100k capacity

The latent is never supervised on object pose — trained only on reconstruction, latent dynamics and future proprioception. Pose is read out afterwards by a probe, for evaluation only.

Caveats

• Single seed per configuration. The 265-vs-56 segdepth gap (1.92 vs 1.96 mm) is smaller than the measured seed σ ≈ 0.15 mm and should be treated as a null result, not a win.
• The state grows 65 → 274 dims and the replay buffer 11 MB → 51 MB at 100k capacity. Given the 0.04 mm difference on the segdepth objective, the 56-d model is the better choice for RL.