realsense-calvin

realsense-calvin

Raw per-timestep CALVIN-format conversion of realsense-converted, generated by dawn/data/realsense/convert_to_raw_calvin.py in the DAWN repo (/home/colligo/Codes/HiVA/DAWN).

Layout

training/episode_0000000.npz ... + training/lang_annotations/auto_lang_ann.npy
validation/episode_0000000.npz ... + validation/lang_annotations/{auto_lang_ann.npy, embeddings.npy}

Per-frame npz keys

• rgb_static (256, 256, 3) uint8 — official CALVIN uses (200, 200, 3); resized to 256x256 instead to match the image_size used by DAWN training configs.
• rgb_gripper (256, 256, 3) uint8 — official CALVIN uses (84, 84, 3); same reason.
• rgb_front (256, 256, 3) uint8 — non-standard bonus view, not part of official CALVIN.
• rel_actions (7,) float32

Intentionally omitted

actions (absolute world-frame TCP pose), robot_obs (proprioception/joint angles), scene_obs (object poses), depth maps, rgb_tactile — none of these exist for the real RealSense recordings, so they are left out rather than fabricated.

Consequence: the stock calvin_agent.datasets.disk_dataset.DiskDataset loader unconditionally reads scene_obs in _load_episode and will raise KeyError on this dataset. Use a custom/lightweight loader that only reads the keys above, or use the dataset_opt_local-style conversion at /home/colligo/Codes/localssd/realsense-dawn with dawn.data.calvin.calvin.CalvinDataset (action_type="actions") instead.

Train/validation episode membership matches realsense-dawn/split_manifest.json.