starvla-qwengr00t-so101-marker-bs8h10

StarVLA-GR00T (QwenGR00T) — SO-101 marker-in-cup

Qwen3-VL-4B backbone (frozen) as System 2 + GR00T-style DiT flow-matching action head (System 1) conditioned on the final VLM hidden state, with a dedicated state encoder, fine-tuned on new_embodiment (97 episodes / 31291 frames, LeRobot v2.1, so101_follower).

• Task: Pick up the marker and place it into the cup, then reset the arm to rest state.
• Checkpoint: checkpoints/steps_30000_pytorch_model.pt (30000 steps)
• Action space: 6-D absolute joint positions — single_arm (5) + gripper (1)
• Cameras: front, wrist — in that order (see below)
• Action horizon: 10

Serving

Needs the starVLA repo including examples/SO101_Marker/train_files/data_registry/data_config.py — the server resolves robot_type: so101_marker through the auto-discovered registry, and fails at startup without it. data_config.py is mirrored in this repo for reference.

huggingface-cli download mimiminsoo/starvla-qwengr00t-so101-marker-bs8h10 --local-dir ./so101_marker_ckpt

python deployment/model_server/server_policy_gr00t_zmq.py \
    --ckpt_path ./so101_marker_ckpt/checkpoints/steps_30000_pytorch_model.pt \
    --port 5555 --use_bf16

framework.qwenvl.base_vlm in config.yaml points at Qwen/Qwen3-VL-4B-Instruct, so the base VLM downloads on first run. Inference needs roughly 11GB of VRAM in bf16.

Isaac Sim / LeIsaac client

Speaks the GR00T N1.6 ZMQ protocol, so the LeIsaac client connects unchanged:

python scripts/evaluation/policy_inference.py \
    --task LeIsaac-SO101-CupStack-v0 \
    --policy_type gr00tn1.6 --policy_host localhost --policy_port 5555

Camera order matters. Training fed front then wrist, while LeIsaac's scene declares wrist before front, so the raw wire dict order is reversed. The server orders images by the DataConfig's video_keys, which fixes this — but only on a starVLA checkout that has that ordering fix in deployment/model_server/gr00t_obs_adapter.py. On an older checkout the two views arrive swapped and the policy misbehaves without erroring.

Training

Simulation teleoperation data collected in Isaac Sim / LeIsaac, so evaluating the policy in the same simulator carries no domain gap. Trained from the base VLM with a randomly initialised action head — no robot pretraining — unlike π0 or GR00T checkpoints that start from large-scale robot pretraining.