skyfull-lookat-chamonix-v1

skyfull-lookat-chamonix-v1

LookAtNet — 4.5k-param MLP that maps a 3D event location to SO-101 arm joint angles for look-at pointing.

Trained in 7 seconds on an M2 MacBook Air on 5,000 simulated pointing episodes over virtual Chamonix terrain (MuJoCo).

Architecture

Input  (3): normalised (x_m, y_m, z_m) — event in arm frame [0,1]
         → Linear(3, 64) + SiLU
         → Linear(64, 64) + SiLU
         → Linear(64, 3)
Output (3): (θ0, θ1, θ2) — joint angles in radians

Quick start

import torch, numpy as np
import torch.nn as nn
from huggingface_hub import hf_hub_download

class LookAtNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(3, 64), nn.SiLU(),
            nn.Linear(64, 64), nn.SiLU(),
            nn.Linear(64, 3),
        )
    def forward(self, x): return self.net(x)

ckpt  = torch.load(hf_hub_download("Ethgar/skyfull-lookat-chamonix-v1", "lookat_chamonix.pt"), weights_only=False)
model = LookAtNet()
model.load_state_dict(ckpt['state_dict'])
model.eval()
norm  = ckpt['norm']

# Event at (x_m=0.20, y_m=0.05, z_m=0.08) in arm frame
xyz   = np.array([0.20, 0.05, 0.08], dtype=np.float32)
x_n   = (xyz - norm['x_min']) / (norm['x_max'] - norm['x_min']).clip(1e-6)
with torch.no_grad():
    y_n = model(torch.from_numpy(x_n).unsqueeze(0)).numpy()[0]
theta = y_n * norm['y_std'] + norm['y_mean']
print(f"θ0={np.degrees(theta[0]):.1f}°  θ1={np.degrees(theta[1]):.1f}°")

Sim-to-real transfer

1. Deploy on real SO-101
2. Point at 3 known reference locations, record joint angles
3. Fit affine: θ_real = A @ θ_sim + b
4. Apply correction at inference — done

Dataset

Ethgar/skyfull-chamonix-lookat-v1