dinov3-readout-threading-d0

Millimetre object pose from frozen DINOv3 — MimicGen threading_d0

Decode metric 3D object position from vision on MimicGen threading_d0, and predict it forward with an action-conditioned world model. Task requirement for reliable insertion is ~4 mm; offset (needle − tripod) is the decision metric, because relative pose is what the controller consumes.

All numbers are mean Euclidean distance in millimetres on held-out episodes — mean_i ‖p_i − t_i‖₂, the average 3D miss distance, not RMSE. Camera geometry: 45° fovy, 0.762 m to the target plane, 2.82 mm per pixel at 224 px; one patch-16 token spans ≈45 mm.

Headline

The best result uses a completely frozen, stock DINOv3-S/16 with no object supervision of any kind — it beats a head trained with ground-truth pose labels.

offset is met on successful trajectories

The pooled number is dominated by failure rollouts (48% of the val set). Per split, one readout:

The entire deficit is tripod localisation once the tripod has been displaced — at which point the threading attempt has already failed.

Two findings that dominate everything else

1. The readout, not the backbone. On identical frozen features, a linear ridge reads 15.6 mm and a learned soft-argmax readout reads 2.1 mm — a 7.4× gap. Encoder choice is irrelevant by comparison: DINOv2-small 3.4, DINOv3-S/16 3.3, VGGT-1B 4.5 mm (matched protocol, 57× parameter range). Finetuning the backbone makes things worse (~50%, with feature-spread collapse).

2. A starved readout manufactures ties. At n=9,600 / 4,000 steps every representation reads ~3.3 mm and looks equivalent. At n=60,000 / 25,000 steps:

SegHead saturates; raw DINOv3 does not. SegHead compresses 602k values into 200k against a seg/depth/RGB objective and discards what a strong readout needs. Its geometry is fully linearly accessible (ridge 2.9 vs head 3.5 — no gap) and therefore capped; DINOv3's is richer but non-linear.

Always verify the probe is saturated before concluding two representations are equivalent.

SegHead is a trade, not an upgrade

SegHead is supervised only on {segmentation, depth, RGB} + proprioception — no object pose — so it is trainable from signals available on real hardware.

World model

Action-conditioned dynamics over the frozen feature map, 8-step horizon, expert split:

4.8× better than assuming nothing moves. For reference, a DINOv2 DINO-WM reaches 0.45, and a world model on pose-supervised GeoHead features reaches 43× worse than copy — its features had collapsed to effective rank 2.4 with 90% dead channels.

Contents

• readout_raw_dinov3/ — the 2.1 mm readout (the headline result). Backbone is stock frozen facebook/dinov3-vits16-pretrain-lvd1689m; only this readout is trained.
• seghead_sup224/ — SegHead, label-free dense supervision at native 224² resolution
• wm_sup224/ — world model on those features (best dynamics result) + its frozen measuring readout
• measurements/ — probe logs

Reproducing

Feed both cameras at 224 px through frozen DINOv3-S/16, take hidden_states[3,6,9,12], drop the first 5 tokens (CLS + 4 registers) to get a 14×14×384 grid per view, then apply the readout: 1×1 conv → 16 heatmaps → spatial softmax → soft-argmax → MLP with z-scored proprioception. Sub-cell precision comes from the soft-argmax expectation: the grid is 45 mm per token and the readout resolves 2.1 mm, ~21× sub-token.

Caveats

• Single seed per configuration; measured spread across 3 seeds is σ ≈ 0.15 mm.
• Only an 8-step horizon was measured for the world model; MPC would use shorter horizons.
• One scene, fixed cameras, one task — this says nothing about transfer, which is where a foundation model would be expected to earn its keep.