Downloads · 30 days
0
sapanostic/cosmos3-edge-so101-pick-place-testtube-helicopter-300eps
cosmos3-edge-so101-pick-place-testtube-helicopter-300eps is a robotics model from sapanostic. Use it for the robotics task on the model card, and read the license before you ship it in a product. It is set up for lerobot. The card lists the license as other.
Supervised fine-tune of nvidia/Cosmos3-Edge into a world-action policy for a single SO-ARM101 arm on the task:
Downloads · 30 days
0
Access
Public
Updated Aug 27, 2026
Repo size
68.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors77.2 GB · 100%
From the Hugging Face model README
Supervised fine-tune of nvidia/Cosmos3-Edge into a
world-action policy for a single SO-ARM101 arm on the task:
"Pick up the test-tube and place it in the holder on the helicopter"
Trained with Rebis-IvLabs/cosmos-framework,
branch experiment/so101-edge, experiment action_policy_so101_edge — the first Edge
action-policy recipe (upstream ships Nano recipes only).
One subfolder per training iteration, each a consolidated bf16 export
(export_model), the same form as the base model:
checkpoint-500, checkpoint-1000, … checkpoint-5000 (every 500 iterations).
checkpoint-5000 is the final and best-scoring one.
| Observation | one RGB frame, third-person and wrist views concatenated horizontally (256×256 each → 256×512) |
| Instruction | the task string above, JSON-formatted (format_prompt_as_json=True) |
| Proprioception | none — the policy is vision + language only |
| Action | 16-step chunk of frame-wise relative end-effector deltas: [dx, dy, dz, rot6d(6), gripper], 10-D, at 15 Hz (≈1.07 s horizon) |
| Also generated | the 17 future frames of the observation (the model is a world-action model; video and action are trained jointly) |
| Normalisation | quantile_rot against so101_native_frame_wise_relative_rot6d.json (bundled in the framework branch) |
| Embodiment | so101 (domain id 25) |
Rotation is packed as the first two columns of R, column-major — Cosmos's convention, not
GR00T's row-major eef_9d. Deltas are body-frame T_i⁻¹·T_{i+1}, so a controller integrates
them from its current pose: T_{i+1} = T_i · Δ_i.
| Data | smr_dataset_so101_pick_place_test_tube_on_the_helicopter_300eps_cosmos3_train (private) — 270 episodes / 77 107 frames, 15 Hz, LeRobot v3.0 |
| Held-out | 30 episodes / 9 111 frames (same split as the GR00T exp-021 baseline) |
| Hardware | 1× NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Schedule | 5000 iterations, ~4.9 h, effective batch 64 (8 samples × grad-accum 8), lr 5e-5 LambdaLinear annealed to 0, warm-up 200 |
| Trainable | moe_gen, time_embedder, vae2llm, llm2vae, action2llm, llm2action, action_modality_embed (action heads at 5× lr) |
| Loss | rectified flow on video and action (loss_scale 10, action_loss_weight 10); 14 → 1.0 |
| Tracking | W&B v64sp2z5 |
Each 16-frame chunk is predicted from its first real frame plus the task string, then integrated from the ground-truth pose at that frame — the re-anchoring a closed-loop controller performs every chunk, so errors do not accumulate across chunks. ADE / FDE are mean / final position error within a chunk.
| checkpoint | ADE (cm) | FDE (cm) | rotation ADE (°) | gripper MAE (%) |
|---|---|---|---|---|
| 500 | 1.64 | 2.60 | 5.3 | 3.7 |
| 1500 | 1.67 | 2.63 | 5.3 | 2.9 |
| 2500 | 1.60 | 2.43 | 4.9 | 2.8 |
| 4000 | 1.54 | 2.33 | 4.7 | 2.5 |
| 5000 | 1.51 | 2.29 | 4.6 | 2.5 |
Gripper predictions track the real grasp (correlation 0.87; 2.34 % MAE against 10.4 % for a constant-mean predictor) but close roughly 6 frames (0.4 s) early, a bias that appears by iteration 2500 and does not improve with further training.
Not yet evaluated closed-loop. These are open-loop trajectory metrics; task success on the real arm is unmeasured.
Serve with the framework's policy server (from the experiment/so101-edge branch):
python -m cosmos_framework.scripts.action_policy_server_libero \
--experiment action_policy_so101_edge \
--experiment-overrides "model.config.tokenizer.vae_path=$WAN_VAE_PATH" \
--checkpoint-path <this repo>/checkpoint-5000 \
--action-normalization quantile_rot \
--action-stats-path cosmos_framework/data/generator/action/normalizer_stats/so101_native_frame_wise_relative_rot6d.json \
--raw-action-dim 10 --fps 15 --port 8000
POST /predict with {"image": <base64 PNG of the concatenated views>, "prompt": <task string>, "domain_name": "so101", "image_size": 256} returns the denormalised 10-D action chunk and the
generated frames.