b601-lingbotva-10k

LingBot-VA post-trained on Seeed B601 — pick red cube

LingBot-VA post-trained on a single-arm Seeed B601 pick-and-place task: pick up the red cube and place it inside the tray. Only the transformer/ is released — the VAE, T5 text encoder and tokenizer are unchanged from the base model and must be fetched separately (see Setup).

⚠️ This checkpoint has not been evaluated. No real-robot rollout and no held-out split were run; the training set has 50 episodes of one task and the run does 800 epochs over it, so expect heavy overfitting to that scene. Treat it as a training artifact, not a policy with known success rate.

Files

Setup

huggingface-cli download robbyant/lingbot-va-base --local-dir /path/to/base
cd <this-checkpoint-dir>
ln -s /path/to/base/vae          vae
ln -s /path/to/base/text_encoder text_encoder
ln -s /path/to/base/tokenizer    tokenizer

Then point a serve/i2va config at this directory:

cfg.wan22_pretrained_model_name_or_path = "<this-checkpoint-dir>"

Copy the layout and action fields from va_b601_train_cfg.py verbatim and add the sampling parameters (untuned starting points, taken from the RoboTwin/UR3 configs):

cfg.guidance_scale = 5
cfg.action_guidance_scale = 1
cfg.num_inference_steps = 25          # video
cfg.video_exec_step = -1
cfg.action_num_inference_steps = 50   # action

Interface

Two cameras, resized to 256 x 256 each (the source is 480 x 640, so the aspect ratio is squashed — do the same at inference), concatenated along width:

The prompt must match the training string character for character:

Pick up the red cube and place it completely inside the tray.

Actions: 7 dims, absolute joint positions in degrees, in leader convention, ordered [shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_yaw, wrist_roll, gripper]. They occupy slots [14, 15, 16, 17, 18, 19, 28] of LingBot-VA's 30-dim action vector; every other slot is unused. One chunk is frame_chunk_size=4 latent frames x action_per_frame=8 = 32 steps at 30 Hz (~1.07 s).

⚠️ observation.state is not in the same convention as the actions. It is the follower reading: shoulder_pan, shoulder_lift, wrist_roll and gripper are sign-flipped, and the gripper uses a different unit (state spans [-270, 0], action [0, 57]). This conversion is not implemented in the LingBot-VA code — a real-robot client has to do it.

Normalization is min-max over the q01/q99 below, mapped to [-1, 1] and clipped at ±1.5, so a value far outside this range saturates:

Training

Post-trained from robbyant/lingbot-va-base (commit 68b7bc1b35da6ddc67ea94c4ceb58d768fbb3f9c).

• Data: 50 episodes / 37,932 frames / 1 task, LeRobot v2.1. Videos pre-encoded with the Wan2.2 VAE at 256 x 256 -> 16 x 16 latents, frame stride 2 over the 30 fps source (15 fps), 84-113 latent frames per episode. One sample is a full episode.
• Recipe: 10,000 optimizer steps, batch size 1 per rank, no gradient accumulation (effective batch 4), AdamW lr 1e-5 (β 0.9/0.95, wd 0.1, 10 warmup steps), grad clip 2.0, bf16, FSDP + activation checkpointing, 10% text-embedding dropout for CFG. Flow-matching loss on video and action jointly, snr_shift 5.0 (video) / 0.05 (action); chunk size 1-4 and attention window 4-64 randomized per step.
• Compute: 4 x H100 NVL, ~17.6 h (~7 s/step).

Not included

• No optimizer/training state — LingBot-VA's save_checkpoint does not write one, so this checkpoint can be used to initialize training but cannot resume a run.
• No VAE / text encoder / tokenizer (identical to the base model, see Setup).
• No empty_emb.pt; that is only used for CFG dropout during training. The inference server encodes the empty string itself.

License

Apache-2.0, inherited from the base model.