vis2tac-latent-sparsh-r38

vis2tac-latent-sparsh-r38

Round 38 SUCCESS: Latent vis2tac diffusion (NOT pixel-space).

Result (BEST ckpt, standalone eval on SSVTP test 46 pairs)

Delta_gen_vs_v = +0.0644 -- PASS Codex GO threshold (+0.025) by 2.6x; PASS Codex predicted upper bound (+0.04) by 1.6x.

In-training validation saw stochastic peaks up to +10.67pp.

Quality (latent space)

• cosine_real_vs_gen: 0.807 (high similarity in DINOv2-base latent space)
• dino_std_gen / dino_std_real: 1.29 (no mode collapse)

Architecture

• Frozen vision encoder: facebook/dinov2-small
• Frozen tactile target encoder: facebook/dinov2-base (Sparsh-style)
• Denoiser: ResidualMLPDenoiser (8 res blocks, hidden=2048, latent=1536)
• DDPM scheduler with v_prediction
• Loss: MSE(v_pred, v_target) + 0.1 x cosine(x0_pred, target)
• EMA 0.999, validation by probe Delta (NOT MSE)

Training

• 30000 optim steps on isaacsim_wy891 H100 GPU 0
• batch_size=256, grad_accum=1, lr=1e-4, warmup=1000
• Manifest: SSVTP10x upsample + HCT video-level split (no leakage)
• ~14h wall-clock incl 60 validations

Why this works (vs pixel-space failure)

Round 37 SSVTP50x pixel-space DDPM (V+gen_T -13pp despite better LPIPS) showed that pixel-level reconstruction trained a visual class-prior shortcut without material-useful representation. Round 38 pivots to latent diffusion -- the MLP denoiser predicts directly into the DINOv2 tactile latent space that the probe consumes, eliminating the pixel-vs-feature mismatch.

Files

• best_unified.pt: best ckpt by Delta_gen_vs_v (peak +10.67pp during training, +6.44pp on standalone re-eval)
• final_unified.pt: last 30k step (overfit at end, V+gen_T -4.2pp)
• tactile_latent_norm.pt: train-set z-score for DINOv2 tactile encoder
• eval_post_train/best/eval_report.{json,md}: standalone eval reports
• eval_post_train/final/eval_report.{json,md}: same for final ckpt

Source code

• path_d/code/train/r38_latent_arch.py: ResidualMLPDenoiser + compute_v_loss
• path_d/code/train/train_v2t_latent_sparsh.py: full trainer
• path_d/code/eval/eval_v2t_latent_probe.py: latent eval

GitHub commit: a831032ccd