EgoSteer-3B-RealMan

EgoSteer-3B-RealMan

EgoSteer-3B-RealMan is the generalist policy from EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos. It is a world-model-enhanced Vision-Language-Action (VLA) policy built on a Qwen3-VL backbone with a flow-matching action expert, post-trained from EgoSteer-3B-Base on real-world demonstrations collected on the RealMan robot.

This is a robot-ready dual-camera (head + chest) checkpoint. For the base model to fine-tune on your own data, see EgoSteer-3B-Base.

• 💻 Code: https://github.com/ego-steer/egosteer

Model Description

Our full-stack system integrates EgoSmith (data pipeline), Robot Stack (deployment), and EgoSteer (policy) to learn from 9.6k hours of large-scale egocentric human videos and facilitate data-efficient real-robot post-training, enabling steerable dexterous manipulation across over 40 tasks alongside few-shot adaptation to complex, long-horizon tasks. EgoSteer-3B-RealMan runs on the RealMan embodiment out of the box and can be further post-trained for other embodiments.

Inputs & Outputs

The policy maps a language instruction plus multi-view RGB and proprioception to a chunk of future actions. Keep these consistent with the bundled config.yaml and normalizer.pkl.

Model Variants

<table> <thead> <tr><th width="30%">Model</th><th width="10%">Parameters</th><th width="60%">Description</th></tr> </thead> <tbody> <tr><td><a href="https://huggingface.co/ego-steer/EgoSteer-3B-Base"><b>EgoSteer-3B-Base</b></a></td><td>3B</td><td>Base EgoSteer model trained on 9.6k hours of egocentric human videos, ready for fine-tuning</td></tr> <tr><td><a href="https://huggingface.co/ego-steer/EgoSteer-3B-RealMan"><b>EgoSteer-3B-RealMan</b></a> (this repo)</td><td>3B</td><td>Generalist post-trained on real-world data collected on the RealMan robot</td></tr> </tbody> </table>

Repository Contents

⚠️ How to Use

This is a custom policy, loaded and run with the EgoSteer codebase.

This is a dual-camera policy, so a few config keys differ from the single-camera base. When serving or evaluating, keep these consistent with the bundled config.yaml:

• data.target_image_size: [480, 640]
• dataset.vla_dataset.load_chest: true (head + chest)
• data.max_vlm_tokens: 2176 — dual-camera uses roughly twice the vision tokens of single-camera; a smaller value truncates the input and fails with a video-token-count mismatch.
• policy.rtc_config.enabled: true — real-time chunking (RTC) for smooth asynchronous closed-loop control; tolerates up to max_delay: 6 steps of inference latency. Disable it for synchronous, blocking action-chunk execution.

Pretrained Backbones

EgoSteer depends on two pretrained backbones; download them ahead of time:

• Qwen/Qwen3-VL-2B-Instruct
• facebook/dinov3-vitl16-pretrain-lvd1689m

Training Data

EgoSteer-3B-RealMan is post-trained from EgoSteer-3B-Base (pretrained on 9.6k hours of egocentric human videos in a unified human-to-robot action space) on real-world dual-camera demonstrations collected on the RealMan robot. The anonymous RealWorld review subset contains 1,920 episodes across 160 tasks; it is a sample, not the full 54,454-episode, 192-hour, 193-task training collection. The anonymous Egocentric review subset contains 40 episodes from 8 sources, with labels only. The 9.6k-hour pretraining figure describes the full corpus, not the downloadable subset. See the paper for details.

Intended Use & Limitations

• Intended use: research on vision-language-action models, world action models, and dexterous manipulation; a robot-ready RealMan generalist and a starting point for post-training.
• Limitations: the policy is tuned to the RealMan dual-camera observation/action format; deploying on a different embodiment requires matching the observation format (camera setup, image size, normalizer) or further post-training. Outputs should be validated for safety before execution on hardware.

License

Released under the Apache 2.0 license.

Acknowledgements

Built on Qwen3-VL and DINOv3.