Downloads · 30 days
0
sapanostic/cosmos3-edge-so101-pick-place-testtube-helicopter-300eps-wrist
cosmos3-edge-so101-pick-place-testtube-helicopter-300eps-wrist is a robotics model from sapanostic. Use it for the robotics task on the model card, and read the license before you ship it in a product. It is set up for lerobot. The card lists the license as other.
Supervised fine-tune of nvidia/Cosmos3-Edge into a world-action policy for a single SO-ARM101 arm on the task:
Downloads · 30 days
0
Access
Public
Updated Aug 30, 2026
Repo size
34.8 GB
Likes
0
Public
Click a slice to open those files.
.safetensors38.6 GB · 99%
From the Hugging Face model README
Supervised fine-tune of nvidia/Cosmos3-Edge into a
world-action policy for a single SO-ARM101 arm on the task:
"Pick up the test-tube and place it in the holder on the helicopter"
Wrist-camera variant of
cosmos3-edge-so101-pick-place-testtube-helicopter-300eps.
Same data, same action space, same base checkpoint — the observation is the wrist view alone
(256×256) instead of the third-person and wrist views concatenated (256×512).
Two reasons that matters: every GR00T experiment on this task (016 onward) is wrist-only, so this is the like-for-like comparison against them; and the live SO-ARM101 rig has only a wrist camera wired, so this is the variant that can be closed-loop evaluated on hardware without adding a second camera.
Trained with Rebis-IvLabs/cosmos-framework,
branch experiment/so101-edge-wrist, experiment action_policy_so101_edge_wrist.
One subfolder per training iteration, each a consolidated bf16 export (export_model), the same
form as the base model:
checkpoint-2000, checkpoint-4000, checkpoint-6000, checkpoint-8000, checkpoint-10000.
checkpoint-10000 is the final and best-scoring one. Checkpoints are 2000 iterations apart
because the training-time DCP checkpoints are ~36 GB each and nothing prunes them.
| Observation | one RGB frame, wrist view only, 256×256 |
| Instruction | the task string above, JSON-formatted (format_prompt_as_json=True), viewpoint tag wrist_view |
| Proprioception | none — the policy is vision + language only |
| Action | 16-step chunk of frame-wise relative end-effector deltas: [dx, dy, dz, rot6d(6), gripper], 10-D, at 15 Hz (≈1.07 s horizon) |
| Also generated | the 17 future frames of the observation (the model is a world-action model; video and action are trained jointly) |
| Normalisation | quantile_rot against so101_native_frame_wise_relative_rot6d.json (bundled in the framework branch) |
| Embodiment | so101 |
Rotation is packed as the first two columns of R, column-major — Cosmos's convention, not
GR00T's row-major eef_9d. Deltas are body-frame T_i⁻¹·T_{i+1}, so a controller integrates
them from its current pose: T_{i+1} = T_i · Δ_i.
| Data | smr_dataset_so101_pick_place_test_tube_on_the_helicopter_300eps_cosmos3_train (private) — 270 episodes / 77 107 frames, 15 Hz, LeRobot v3.0 |
| Held-out | 30 episodes / 9 111 frames (same split as the GR00T exp-021 baseline) |
| Hardware | 1× NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Schedule | 10 000 iterations, 11.0 h, effective batch 64 (8 samples × grad-accum 8), lr 5e-5, warm-up 200, cycle length 10 000 |
| Trainable | moe_gen, time_embedder, vae2llm, llm2vae, action2llm, llm2action, action_modality_embed (action heads at 5× lr) |
| Loss | rectified flow on video and action (loss_scale 10, action_loss_weight 10) |
| Tracking | W&B 92qiu1ug |
Training loss, mean over each 1000-iteration window:
| iters | 1–1k | 1–2k | 2–3k | 3–4k | 4–5k | 5–6k | 6–7k | 7–8k | 8–9k | 9–10k |
|---|---|---|---|---|---|---|---|---|---|---|
| loss | 2.730 | 2.401 | 2.201 | 2.017 | 1.882 | 1.786 | 1.702 | 1.647 | 1.558 | 1.523 |
Each 16-frame chunk is predicted from its first real frame plus the task string, then integrated from the ground-truth pose at that frame — the re-anchoring a closed-loop controller performs every chunk, so errors do not accumulate across chunks. ADE / FDE are mean / final position error within a chunk.
| checkpoint | ADE (cm) | FDE (cm) | rotation ADE (°) | gripper MAE (%) | action MSE |
|---|---|---|---|---|---|
| 2000 | 1.77 | 2.61 | 5.25 | 2.68 | 3.72e-4 |
| 4000 | 1.64 | 2.39 | 4.80 | 2.62 | 4.07e-4 |
| 6000 | 1.60 | 2.30 | 4.74 | 2.39 | 3.87e-4 |
| 8000 | 1.55 | 2.25 | 4.72 | 2.27 | 3.93e-4 |
| 10000 | 1.49 | 2.18 | 4.54 | 2.20 | 3.78e-4 |
Per-episode JSON and rollout videos for every evaluated checkpoint are in eval/.
Both the training loss and every eval metric were still improving at iteration 10 000 — the run was stopped by its iteration budget, not by convergence. A longer cycle is the obvious next experiment.
Against the concat-view run at its own final checkpoint (5000 iterations: ADE 1.51 cm, FDE 2.29 cm, rot 4.6°, gripper 2.5%), wrist-only at 10 000 iterations is marginally better on every metric — but the two ran different iteration budgets, so this is not a controlled comparison of the observation format.
Not yet evaluated closed-loop. These are open-loop trajectory metrics; task success on the real arm is unmeasured.
Serve with the framework's policy server (from the experiment/so101-edge-wrist branch):
python -m cosmos_framework.scripts.action_policy_server_libero \
--experiment action_policy_so101_edge_wrist \
--experiment-overrides "model.config.tokenizer.vae_path=$WAN_VAE_PATH" \
--checkpoint-path <this repo>/checkpoint-10000 \
--action-normalization quantile_rot \
--action-stats-path cosmos_framework/data/generator/action/normalizer_stats/so101_native_frame_wise_relative_rot6d.json \
--raw-action-dim 10 --fps 15 --port 8000
POST /predict with {"image": <base64 PNG of the wrist frame>, "prompt": <task string>, "domain_name": "so101", "image_size": 256} returns the denormalised 10-D action chunk and the
generated frames.
These checkpoints were exported with two aliases added to the exporter's public-config table
(cosmos_framework/inference/common/public_model_config.py), which upstream's 2026-08-19 release
left unregistered:
"projects.cosmos3.vfm.configs.base.defaults.flex_attention.FlexAttentionConfig": "flex_attention_config",
"projects.cosmos3.vfm.configs.base.defaults.flex_attention.FlexAttentionMaskConfig": "flex_attention_mask_config",
"projects.cosmos3.vfm.models.utils.load_balancing_stats.LBLConfig": "lbl_config",
Without them export_model raises No public alias registered for type path: ..., and a framework
copy lacking them cannot resolve flex_attention_config / flex_attention_mask_config when
reading these config.json files. The experiment/so101-edge-wrist branch carries the patch.