dreamzero

T-Rex Dataset

A large-scale, tactile-reactive bimanual manipulation dataset, collected via teleoperation on a Dexmate Vega-1 robot with two Sharpa Wave dexterous hands. Stored as a LeRobotDataset v3.0.

🌐 Project Page · ✍️ Paper (arXiv) · 💻 Code (T-Rex) · 🚀 Dataset Quickstart · 📓 Colab notebook

<p align="center"> <img src="assets/dataset_overview.png" width="100%"> <br> <em>One episode from each of 20 motor primitives (head-camera view, cropped to the workspace), each with a different object.</em> </p> <p align="center"> <img src="assets/hardware_setup.png" width="100%"> <br> <em>Teleoperation setup: Manus gloves + VIVE trackers drive the bimanual Dexmate Vega-1 with two Sharpa Wave hands; observations come from a head-mounted ZED X Mini camera and two wide-view ZED X One S wrist cameras.</em> </p>

At a glance

Composition

<p align="center"> <img src="assets/dataset_analysis.png" width="100%"> <br> <em>Object categories, episodes per motor primitive, and per-object episode counts.</em> </p>

Collection setup

The full teleoperation/hardware stack used to collect the dataset is open-sourced in hardware_code/ in the T-Rex repo.

• Robot. The Dexmate Vega-1 is a dual-arm mobile robot with 7 actuated joints per arm, here equipped with two Sharpa Wave dexterous hands. During collection the wheels, torso, and head joints are fixed; only the 14 arm joints and the two hands are actuated.
• Cameras. A head-mounted ZED X Mini stereo camera (its left monocular RGB stream is recorded) plus two wide-view ZED X One S monocular RGB cameras mounted on the wrists, posed so the head camera covers the full reachable workspace while the wrist cameras keep the fingers visible without significant palm occlusion. All three streams are recorded at 640×360.
• Tactile. Each hand carries five fingertip tactile sensors. Per sensor, the dataset stores the raw sensor image and the estimated deformation map (both as video), and the estimated 6-axis net wrench (observation.tactile_force).
• Teleoperation. Manus gloves capture fingertip positions relative to the hand base, retargeted to the Sharpa Wave hands with the manufacturer's differential-inverse-kinematics package (Pinocchio + CasADi). Two VIVE trackers provide SE(3) wrist poses, converted to arm joint commands via differential inverse kinematics (Pink), low-pass filtered, and tracked by the manufacturer's low-level cascade PID controller. A 30 Hz high-level thread records observations and joint-space targets while asynchronously updating a 300 Hz low-level control thread — the dataset's 30 fps matches the high-level loop, and action holds its 30 Hz joint-space targets.

Per-frame features

Finger order is thumb, index, middle, ring, pinky. Full per-dimension joint names are in meta/info.json (features[*].names).

Per-episode metadata

meta/episodes/*.parquet carries language and task labels per episode, in addition to the standard LeRobot fields (episode_index, tasks, length, per-feature stats, video pointers):

Tactile video encoding (read before decoding)

The raw and deformation tactile videos are stored losslessly (libx264 -qp 0) because their pixel values are physically meaningful (raw sensor images and deformation maps). They are grayscale — the signal lives entirely in the luma (Y) plane — and use full-range yuvj420p, so values span the full 0–255 with no range conversion. Decode the luma plane to recover the original uint8 images exactly, e.g. frame.to_ndarray(format="gray") in PyAV.

No web thumbnails for tactile. Lossless H.264 forces the High 4:4:4 Predictive profile, which most browsers and the Hugging Face preview cannot decode — so the tactile videos do not show thumbnails on the dataset page. This is expected; decode them locally (ffmpeg / PyAV / torchcodec). The RGB cameras use standard yuv420p (limited range, BT.709) and preview normally.

Usage

With LeRobot

from lerobot.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("zekaiwang/trex_dataset")
frame = ds[0]  # dict of tensors: observation.state, action, observation.tactile_force, images...

Stream individual episodes (no full download)

The T-Rex Quick Start companion repo browses, inspects, and replays single episodes without downloading the full dataset, and includes a notebook you can open directly in Colab.

Dataset viewer

The Hugging Face table viewer is manually configured (the configs block above) with two views:

• frames (default) — the raw per-frame observation.state, action, and observation.tactile_force arrays (plus index columns). This is the actual data, and what datasets.load_dataset("zekaiwang/trex_dataset") returns by default.
• episodes — one row per episode with its language caption, motor_primitive, object, and target: a readable table of contents. Switch to it with the config dropdown, or load it with datasets.load_dataset("zekaiwang/trex_dataset", "episodes").

Videos are stored as separate .mp4 files referenced by timestamp pointers, so they are not shown in the table viewer — load them via LeRobot or the quick-start tools. For interactive video playback, use the LeRobot dataset visualizer.

Layout

data/chunk-000/file-*.parquet     per-frame state / action / tactile_force (+ index columns)
videos/<key>/chunk-*/file-*.mp4   23 video streams (3 RGB + 20 tactile)
episodes_preview.parquet          curated per-episode labels for the web viewer (see "Dataset viewer")
meta/info.json                    features, shapes, fps, codebase_version
meta/episodes/*.parquet           per-episode metadata + stats + video pointers
meta/tasks.parquet                task (caption) table
meta/stats.json                   global feature statistics

Citation

If you find the T-Rex Dataset useful, please cite:

@misc{trex2026,
  title={T-Rex: Tactile-Reactive Dexterous Manipulation}, 
  author={Dantong Niu and Zhuoyang Liu and Zekai Wang and Boning Shao and Zhao-Heng Yin and Anirudh Pai and Yuvan Sharma and Stefano Saravalle and Ruijie Zheng and Jing Wang and Ryan Punamiya and Mengda Xu and Yuqi Xie and Yunfan Jiang and Letian Fu and Konstantinos Kallidromitis and Matteo Gioia and Junyi Zhang and Jiaxin Ge and Haiwen Feng and Fabio Galasso and Wei Zhan and David M. Chan and Yutong Bai and Roei Herzig and Jiahui Lei and Fei-Fei Li and Ken Goldberg and Jitendra Malik and Pieter Abbeel and Yuke Zhu and Danfei Xu and Jim Fan and Trevor Darrell},
  year={2026},
  eprint={2606.17055},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2606.17055}, 
}

License

Released under the MIT License © 2026 The Regents of the University of California. See LICENSE.