EgoSteer-3B-Base

EgoSteer-3B-Base

EgoSteer-3B-Base is the base world-model-enhanced Vision-Language-Action (VLA) policy from EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos. It is built on a Qwen3-VL backbone with a flow-matching action expert and a DINOv3 latent future-prediction objective used only during training, and learns a unified human-to-robot action space from large-scale egocentric human videos.

This is the pretrained base checkpoint. For a robot-ready generalist, see EgoSteer-3B-RealMan.

• 🌐 Project page: https://egosteer.github.io/
• 📄 Paper: https://arxiv.org/abs/2607.09701
• 💻 Code: https://github.com/egosteer/egosteer

Model Description

Our full-stack system integrates EgoSmith (data pipeline), Robot Stack (deployment), and EgoSteer (policy) to learn from 9.6k hours of large-scale egocentric human videos and facilitate data-efficient real-robot post-training, enabling steerable dexterous manipulation across over 40 tasks alongside few-shot adaptation to complex, long-horizon tasks. EgoSteer-3B-Base is the pretrained base and a starting point for data-efficient post-training on real robots.

Inputs & Outputs

The policy maps a language instruction plus RGB and proprioception to a chunk of future actions. Keep these consistent with the bundled config.yaml and normalizer.pkl.

Model Variants

<table> <thead> <tr><th width="30%">Model</th><th width="10%">Parameters</th><th width="60%">Description</th></tr> </thead> <tbody> <tr><td><a href="https://huggingface.co/EgoSteer/EgoSteer-3B-Base"><b>EgoSteer-3B-Base</b></a> (this repo)</td><td>3B</td><td>Base EgoSteer model trained on 9.6k hours of egocentric human videos, ready for fine-tuning</td></tr> <tr><td><a href="https://huggingface.co/EgoSteer/EgoSteer-3B-RealMan"><b>EgoSteer-3B-RealMan</b></a></td><td>3B</td><td>Generalist post-trained on real-world data collected on the RealMan robot</td></tr> </tbody> </table>

Repository Contents

⚠️ How to Use

This is a custom policy, loaded and run with the EgoSteer codebase using the bundled config.yaml to rebuild the network.

Pretrained Backbones

EgoSteer depends on two pretrained backbones; download them ahead of time:

• Qwen/Qwen3-VL-2B-Instruct
• facebook/dinov3-vitl16-pretrain-lvd1689m

Training Data

EgoSteer-3B-Base is pretrained on 9.6k hours of egocentric human videos, framed in a unified human-to-robot action space (wrist poses + fingertip keypoints). The EgoSteer-3B-RealMan variant is additionally post-trained on real-world demonstrations collected on the RealMan robot. See the paper and project page for details.

Intended Use & Limitations

• Intended use: research on vision-language-action models, world action models, and dexterous manipulation; a starting point for fine-tuning on your own embodiment.
• Limitations: the base checkpoint is not tuned to any single robot's control loop; real-robot deployment requires post-training and matching the observation/action format (camera setup, normalizer) the policy was trained with. Outputs should be validated for safety before execution on hardware.

Citation

@misc{zhong2026egosteerfullstacksteerabledexterous,
  title={EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos},
  author={Yifan Zhong and Zhang Chen and Tianrui Guan and Fanlian Zeng and Yuyao Ye and Tianjia He and Ka Nam Lui and Jiayi Li and Tingrui Zhang and Ruilin Yan and Xinhao Ji and Guangyu Zhao and Wenjie Lou and Jiayuan Zhang and Yuanpei Chen and Yaodong Yang},
  year={2026},
  eprint={2607.09701},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2607.09701},
}

License

Released under the Apache 2.0 license.

Acknowledgements

Built on Qwen3-VL and DINOv3.