Downloads · 30 days
7
41% of all-time downloads
EgoSteer-3B-Base
ego-steer/EgoSteer-3B-Base
EgoSteer-3B-Base is a robotics model from ego-steer. Use it for the robotics task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
EgoSteer-3B-Base is the base world-model-enhanced Vision-Language-Action (VLA) policy from EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos. It is built on a Qwen3-VL backb…
All-time downloads
17
Public
Repo size
5.6 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt5.6 GB · 100%
At a glance
- Task
- Robotics
- License
- apache-2.0
- Access
- Public
- Created
- Sep 14, 2026
- Updated
- Sep 26, 2026
- SHA
- df62cf68
Base models
Model card
From the Hugging Face model README
EgoSteer-3B-Base
EgoSteer-3B-Base is the base world-model-enhanced Vision-Language-Action (VLA) policy from EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos. It is built on a Qwen3-VL backbone with a flow-matching action expert and a DINOv3 latent future-prediction objective used only during training, and learns a unified human-to-robot action space from large-scale egocentric human videos.
This is the pretrained base checkpoint. For a robot-ready generalist, see EgoSteer-3B-RealMan.
Model Description
Our full-stack system integrates EgoSmith (data pipeline), Robot Stack (deployment), and EgoSteer (policy) to learn from 9.6k hours of large-scale egocentric human videos and facilitate data-efficient real-robot post-training, enabling steerable dexterous manipulation across over 40 tasks alongside few-shot adaptation to complex, long-horizon tasks. EgoSteer-3B-Base is the pretrained base and a starting point for data-efficient post-training on real robots.
| Component | Description |
|---|---|
| Backbone | Qwen3-VL-2B-Instruct |
| Action expert | Flow-matching (DiT / AdaLN) expert reusing the backbone KV prefix |
| World model expert | Regresses future-frame features from a frozen DINOv3 ViT-L/16 teacher, training only |
| Action space | Unified human-to-robot space based on wrist poses and fingertip keypoints |
| Total parameters | ~3B |
Inputs & Outputs
The policy maps a language instruction plus RGB and proprioception to a chunk of future actions.
Keep these consistent with the bundled config.yaml and normalizer.pkl.
| Specification | |
|---|---|
| Instruction | A natural-language task description |
| Cameras | Single RGB — head, 384 × 384, 6-frame history (stride 30 over a 30 fps base) |
| Intrinsics | Per-camera [fx, fy, cx, cy] (head) — required. By default rendered into the VLM prompt as text (camera_intrinsic_mode: text); rescaled to match the resized image |
| Proprioception (state) | 48-D: bimanual wrist poses (2 × [3 translation + 6D rotation] = 18), expressed in the camera frame + fingertip keypoints (2 hands × 5 fingertips × 3D = 30), expressed in wrist frame, 6-frame history |
| Action output | 48-D relative action in the same layout as the state, and relative to the current state, predicted as a 32-step action chunk |
| Normalization | State/action normalized with the bundled normalizer.pkl (relative action space) — required for inference and fine-tuning |
Model Variants
<table> <thead> <tr><th width="30%">Model</th><th width="10%">Parameters</th><th width="60%">Description</th></tr> </thead> <tbody> <tr><td><a href="https://huggingface.co/ego-steer/EgoSteer-3B-Base"><b>EgoSteer-3B-Base</b></a> (this repo)</td><td>3B</td><td>Base EgoSteer model trained on 9.6k hours of egocentric human videos, ready for fine-tuning</td></tr> <tr><td><a href="https://huggingface.co/ego-steer/EgoSteer-3B-RealMan"><b>EgoSteer-3B-RealMan</b></a></td><td>3B</td><td>Generalist post-trained on real-world data collected on the RealMan robot</td></tr> </tbody> </table>Repository Contents
| File | Description |
|---|---|
model_bf16.pt | bf16 model weights |
config.yaml | Pretraining config, for rebuilding the network at eval and inference. Fine-tuning uses your own config, weights only |
normalizer-relative-10k-pretrain/normalizer.pkl | State/action normalizer calculated on 9.6k hours of egocentric human videos in relative action space |
⚠️ How to Use
This is a custom policy, loaded and run with the
EgoSteer codebase using the bundled
config.yaml to rebuild the network.
Pretrained Backbones
EgoSteer depends on two pretrained backbones; download them ahead of time:
Training Data
EgoSteer-3B-Base is pretrained on 9.6k hours of egocentric human videos, framed in a unified human-to-robot action space (wrist poses + fingertip keypoints). The 9.6k-hour figure describes the full pretraining corpus, not the data downloadable from this account. For anonymous review, Egocentric-subset provides 40 episodes from 8 source datasets, with labels only and no source images or videos. The EgoSteer-3B-RealMan variant is additionally post-trained on real-world demonstrations collected on the RealMan robot. See the paper for details.
Intended Use & Limitations
- Intended use: research on vision-language-action models, world action models, and dexterous manipulation; a starting point for fine-tuning on your own embodiment.
- Limitations: the base checkpoint is not tuned to any single robot's control loop; real-robot deployment requires post-training and matching the observation/action format (camera setup, normalizer) the policy was trained with. Outputs should be validated for safety before execution on hardware.
License
Released under the Apache 2.0 license.
Acknowledgements
- Task
- Robotics
- License
- apache-2.0
- Languages
- en
- Created
- Sep 14, 2026
- Updated
- Sep 26, 2026