lingbot-va-mot

lingbot-va-mot

Converted, not fine-tuned, base checkpoint for WanMoTTransformer3DModel — the Mixture-of-Transformers (MoT) video-action architecture from joliachen/lingbot-va (model_mot.py). 10.0B total parameters. Produced from the shared-backbone lingbot-va-base (WanTransformer3DModel) checkpoint via convert_to_mot.py.

A fine-tuned derivative of this checkpoint (LIBERO-Goal post-train) is available at ZhuoranChen/lingbot-va-mot-posttrain-libero-goal-gradaccum43.

How the action-stream weights were initialized

The action stream is losslessly cloned from the corresponding video-stream weights at conversion time — no interpolation, no random initialization, no dimension change (both streams are the same width, see below):

applied identically for all 30 transformer blocks. action_embedder, action_proj_out, and condition_embedder_action.* were already present in the base checkpoint and are kept as-is (not cloned). patch_embedding.* from the base checkpoint is dropped (unused by the MoT model). At this conversion step — before any post-training — the action stream is a byte-identical copy of the video stream; the two only diverge once fine-tuning starts.

Architecture: 3072-dim action stream, not 768 (deviation from the lingbot-va paper)

The lingbot-va paper describes the action stream operating through a narrower 768-dim bottleneck (raw 30-dim action → 768 → … → 768 → 30). This checkpoint's architecture has no such bottleneck: the action stream runs at the full video-stream width, d_v = 3072, throughout:

• action_embedder: Linear(30 → 3072) (weight shape [3072, 30])
• action_proj_out: Linear(3072 → 30) (weight shape [30, 3072])
• condition_embedder_action.* (text/time embedders, time projection): all at 3072, shape-parallel to the video-stream condition_embedder
• every per-block action module (action_attn1/2, action_norm2, action_ffn, action_scale_shift_table) is shape-identical to its video-stream counterpart at 3072

Net effect: the action stream holds ~5.0B of the model's 10.0B params — full width/capacity parity with the video stream, rather than the paper's narrower 768-dim bottleneck design.

Config

Repo contents

Full standalone layout, ready to load directly:

• transformer/ — this checkpoint's 10.0B-param MoT weights
• text_encoder/ — google/umt5-xxl (stock, unmodified)
• tokenizer/ — matching T5 tokenizer
• vae/ — Wan2.1 AutoencoderKLWan (stock, unmodified)