dinov3-geohead-threading-d0

GeoHead — millimeter geometry from frozen DINOv3, on MimicGen threading_d0

Decode metric 3D object position from vision features on MimicGen threading_d0. Task requirement for reliable insertion is ~4 mm; the offset target (needle − tripod) is the decision metric, because relative pose is what the controller consumes.

All numbers are mean Euclidean distance in millimetres on held-out episodes — that is, mean_i ||p_i − t_i||₂, the average 3D miss distance, not RMSE (an earlier revision of this card called it RMSE; the code has always computed the mean of the per-sample norm). RMSE would be the larger sqrt(mean_i ||p_i − t_i||²). Readout is a dual-form ridge (closed form, λ swept, best-val) unless stated.

One caveat on offset (needle − tripod): roughly half its error comes from a small set of frames in failure rollouts where the tripod has been knocked over and lies on the floor (z ≈ 43 mm against a 894 mm nominal height). Those frames are 1.6% of the evaluation set but are far outside the training distribution of tripod positions. The needle number is not affected by this. Camera geometry: 45° fovy, 0.762 m to the target plane, 631 mm view height → 2.82 mm per pixel at 224 px; one patch-16 token spans ≈39.5 mm.

Headline

The geometry was never missing from frozen DINOv3 features — it was not linearly decodable. A 3.8M-parameter decoder recovers 2.4 mm on needle and 5.4 mm on offset at matched sample size, and keeps improving with data (~1.15 mm per 2.5× data, no saturation observed).

The encoder already resolves 1 mm

Decoder removed from the loop entirely: replay a stored sim_state, re-render, displace one object by a known amount, re-render, and measure the feature change. The renderer is deterministic, so re-rendering an identical state gives a bit-identical feature vector — a true zero noise floor.

A 1 mm displacement produces a large, consistent response. Every remaining millimetre is a decoding problem, not an information limit. Caveat: this measures signal in isolation, not signal against nuisance variation (arm motion, occlusion), so "present" ≠ "easily recoverable".

What does NOT help (all measured, all negative)

Scaling the vision side is closed. Changing the readout is what moved the number.

Ablating the head: it is the readout, not the fine feature map

With no upsampling at all most of the gain is already there; a parameter-free bilinear upsample matches the learned one. Replacing soft-argmax with a flat MLP is catastrophic (201 mm). The active ingredient is the soft-argmax spatial prior, not the fine map and not nonlinearity per se.

Mechanism: soft-argmax returns the expected position of a spatial distribution, so the output moves continuously as probability mass shifts and can localize within a patch. A global linear map cannot express that. Measured: the head reads out at 6.6 mm from tokens spanning 39.5 mm.

Readout capacity: already saturated

Non-monotonic in k (32 worse than both neighbours) ⇒ single-run noise is ≈ ±0.5 mm. Quadrupling keypoints to 384 scalars does nothing for offset. k=16, width=512 is at the optimum.

Methodological warnings

1. Sub-1 mm differences from single runs are noise. The k-sweep's non-monotonicity puts the noise floor near ±0.5 mm. Only two effects clearly exceed it: decoder-vs-linear (2.4 mm) and data scaling (2.3 mm). Resolution (0.6 mm), pre-LN vs post-LN (0.5 mm) and a one-off DPT win (1.0 mm, did not replicate) are all within noise. Seeds are needed before trusting any of them.
2. Offset error exceeding its components is expected, not an anomaly. Subtraction adds independent errors in quadrature. Measured/predicted ratio is a stable 1.18 across encoders (e.g. needle 12.0 ⊕ tripod 11.1 → 16.3 predicted vs 19.3 measured). An earlier framing of this as "error decorrelation" was based on inverted algebra.
3. Sample size dominates the encoder effects. Same encoder and grid: 13.6 mm at n=3,840 vs 11.3 mm at n=9,600. That 2.3 mm data effect exceeds the entire S→B→L capacity sweep (0.5 mm). All probes ran at 1.5–3.8% of the 251,811 available windows.
4. A DPT-style multi-layer head was tried and failed (9.5→8.4 at 9.6k, then 14.9 and 12.6 at 25k/60k — worse with more data). Diagnosed: its 112×112 output gives a softmax over 12,544 bins, where soft-argmax returns ≈ image centre and the gradient is ~16× weaker than at 28×28. This is a flaw in pairing DPT with a global soft-argmax, not a verdict on multi-layer fusion.
5. Supervision is privileged simulator pose. On real hardware EEF is free (forward kinematics); object pose is not. The deployable form is metric correspondence from calibrated depth. Read the object rows as an upper bound.
6. Best-val checkpoint selection makes reported numbers mildly optimistic; validation holds out episodes but frames within an episode are correlated. Generalization to a different placement distribution is untested.

Architecture

RGB 224²  →  facebook/dinov3-vits16-pretrain-lvd1689m  (FROZEN, 22M)  →  14×14 × 384
   → LayerNorm+Linear 384→256 → Conv3×3 → Conv3×3+PixelShuffle×2 → Conv3×3
   → fine feature map (256, 28, 28)            ← consumed by the world model
   → 1×1 conv → 16 heatmaps → spatial softmax → soft-argmax (u,v,conf) per camera
   → MLP 96→512→256→12  =  (eef, needle, tripod, offset) × xyz

GeoHead 3.83M trainable · world-model predictor 2.51M · trunk frozen. Two cameras (agentview, robot0_eye_in_hand). Patch tokens start at index 5 (CLS + 4 registers).

Status

Weights for the full 150k-window training run land in geohead/, wm_residual/ and wm_absolute/ when the overnight pipeline completes. This card documents the measurement campaign that selected the architecture.