tactile-1-v1

Tactile-1 v1: Cross-Modal Visual-Tactile World Model

A cross-modal predictor that predicts tactile embeddings from visual observations using frozen pretrained encoders.

Architecture

Training

• Dataset: YCB-Slide (50 trajectories, 60K frames, paired RGB + GelSight tactile)
• Split: Leave-2-Objects-Out (L2OO) — holdout: tomato soup can, hammer
• Hardware: NVIDIA A100 SXM 80GB, RunPod
• Time: 18 minutes (11 epochs, early stopped at epoch 10)
• Batch size: 128, num_workers=2, bfloat16 AMP
• Optimizer: AdamW, lr=3e-4, weight_decay=0.01, cosine warmup (2000 steps)
• Loss: Smooth L1 on LayerNorm-normalized tactile targets

Results

The model converged rapidly — most learning happened in epoch 0 (loss: 0.52 → 0.011).

Usage

import torch
from tactile_jepa.encoders import FrozenDINOv2, FrozenSparsh
from tactile_jepa.architectures import CrossAttnPredictor, ActionEncoder
from tactile_jepa.cross_modal_jepa import CrossModalJEPA
from tactile_jepa.losses import CrossModalLoss

# Build model
model = CrossModalJEPA(
    visual_encoder=FrozenDINOv2(),
    tactile_encoder=FrozenSparsh(),
    predictor=CrossAttnPredictor(dim=384, depth=6, num_heads=6, num_queries=16,
                                  visual_dim=384, output_dim=768, dropout=0.1),
    action_encoder=ActionEncoder(action_dim=7, hidden_dim=384),
    predcost=CrossModalLoss(target_dim=768),
).to("cuda")

# Load checkpoint
ckpt = torch.load("best.pth.tar", map_location="cuda", weights_only=False)
model.load_state_dict(ckpt["model_state_dict"], strict=False)
model.eval()

# Predict tactile from visual input
# visual_obs: [B, T, 3, 224, 224] ImageNet-normalized
predicted_tactile = model.predict_tactile(visual_obs)  # [B, N_queries, 768]

Files

• best.pth.tar — Best checkpoint (epoch 0, val_pred=0.0108), 382MB
• config.yaml — Training configuration

Limitations

• Trained only on YCB-Slide (50 trajectories, single sensor type)
• Actions not yet enabled (Phase A only — Phase B with IDM is planned)
• No cross-modal retrieval or effective rank evaluation yet
• Sparsh encoder uses CC-BY-NC license (Meta FAIR) — not for commercial use

Citation

@software{tactile_jepa_2026,
  title={Tactile-JEPA: Cross-Modal Visual-Tactile World Model},
  author={Julian Saks},
  year={2026},
  url={https://github.com/kingulio8238/tactile_jepa}
}