Isaac-0.5

<!-- Add the final license and pipeline_tag before publication. -->

Isaac 0.5 by Perceptron

Introducing Isaac 0.5, our open foundation model for robot learning.

Isaac 0.5 brings multimodal video understanding, embodied reasoning, spatial grounding, task-progress estimation, and robot control into one 36-billion-parameter sparse model. It reads images, video, language instructions, robot state, and previous actions, then produces text, normalized coordinates, task-state outputs, or robot actions.

To our knowledge, Isaac 0.5 is the first open model operating at the frontier of multimodal video understanding, embodied reasoning, and robot control.

Read the paper · Download the weights (COMING SOON) · View the code

Extending the frontier of open robot learning

Isaac 0.5 is trained on more than 35 robot systems, 100,000 hours of robot experience, one million hours of general video, and three trillion multimodal tokens. Video understanding, spatial grounding, task progress, future-percept prediction, and robot action are co-trained from the beginning on one shared backbone.

Teams can fine-tune Isaac as a robot policy or use its visual outputs inside a planner, controller, or data engine. The release includes base and action checkpoints, action-training and inference code, LeRobot integration, a reference policy server, evaluation tools, and the manifests needed to reproduce the model's data and checkpoint interfaces.

What's new in Isaac 0.5

• A scaling law for video and robot data: Isaac measures how general video, egocentric video, UMI data, and teleoperation data trade off as training scales.
• Semantic world modeling: The model predicts compact future percepts that capture task-relevant changes such as object motion, contact, grasp state, spatial relationships, and task progress.
• Unified perception, reasoning, and control: One shared sparse backbone supports video understanding, pointing, tracking, task-state estimation, and robot action generation.
• Continuous and discrete action interfaces: Isaac supports continuous control through a dedicated Flow expert and diffusion transformer, plus discrete control through a 2,048-token FAST action vocabulary.
• Real-time closed-loop control: Isaac predicts the next action chunk while the current chunk is still executing, using the latest observation and previously issued commands.
• Open training and deployment stack: The release includes model weights, training code, inference code, LeRobot integration, a reference policy server, evaluation code, and reproduction manifests.

A scaling law for video and robot experience

Isaac 0.5 establishes a scaling law for choosing the mix of broad video and expensive robot demonstrations.

We trained Isaac across a grid of general-video, egocentric-video, UMI, and teleoperation mixtures, then measured action-prediction loss on robot trajectories held out from training. The resulting contours map data mixtures that reach the same held-out action loss.

At a target action loss of 2.50:

• a model trained on 1,000 hours of general video requires about 5,900 hours of teleoperation;
• a model trained on one million hours of general video requires about 28 hours of teleoperation.

This is a 210× reduction in required teleoperation. The result gives teams a practical way to decide when another order of magnitude of video is valuable and when collecting more robot experience matters more.

General video contributes broad coverage of objects, activities, environments, and tasks. Egocentric video adds close-up, task-relevant views of hands, motion, and contact. UMI data connects visual knowledge to manipulation. Teleoperation grounds that knowledge in a specific embodiment and action space. Isaac trains on these sources together so each shapes the representation used for control.

Semantic world modeling

Isaac learns from video by predicting future percepts: compact representations of changes that matter for a task. These include which object moved, how objects are related, whether contact occurred, whether a grasp held, and which stage of a task has been reached.

Targets are derived from later frames, including general video without robot action labels. Given observations through time t, Isaac predicts the percept at a later time t + Δ:

$$ L_{\text{percept}} = \mathbb{E}\left[\ell\left(g_\theta(o_{\leq t}), z_{t+\Delta}\right)\right] $$

Here, o≤t is the observation history and z(t + Δ) is the future percept. This objective trains the same backbone Isaac uses to produce actions.

Model architecture

Isaac uses a 36B Qwen-family vision-language backbone with sparse experts. Text, vision, time, robot state, and action history enter one shared sequence.

The autoregressive interface produces text, normalized coordinates, task-state outputs, and discrete actions. Discrete control uses a separate vocabulary of 2,048 FAST action tokens. For continuous control, backbone states condition a dedicated Flow expert and 36-block diffusion transformer that generates an action chunk.

Each mixture-of-experts layer gives every token access to 256 learned experts and a null route. A token may use zero to eight routed experts, while a shared expert and residual path remain active. This lets Isaac vary routed compute across visual, language, state, and action tokens. The checkpoint contains 36B parameters in total.

Performance benchmarks

We evaluate the same Isaac checkpoints across multimodal video understanding, spatial grounding, held-out action prediction, and closed-loop robot control.

Up and to the left is better. Ringed Isaac 0.5 results sit on the frontier; hollow markers are dominated.

Where Isaac sits among open models

Building with Isaac in the open

Isaac 0.5 can be fine-tuned on new demonstrations, used as a visual backbone inside a larger embodied system, or deployed as an action policy through LeRobot or our reference server.

The release includes:

• base and action-capable weights;
• continuous Flow and discrete FAST action configurations;
• action training and fine-tuning code;
• text, pointing, tracking, and task-state output schemas;
• LeRobot integration and a reference policy server;
• evaluation code, task definitions, and rollout manifests;
• checkpoint, data, and model-I/O manifests;
• the technical report, model card, and reproduction guide.

Resources

• Weights (COMING SOON): Hugging Face
• Code: GitHub
• Technical report: Read the paper

Open models are essential to robotics progress. We are releasing Isaac 0.5 with its weights, code, interfaces, benchmarks, and manifests so others can inspect, reproduce, and extend the work.

For help deploying Isaac on your infrastructure, contact [email protected].