weaver-robolab-pi05-ft

WEAVER fine-tuned on Pi0.5 RoboLab demos — model card & usage

A fine-tune of the released WEAVER action-conditioned multi-view world model on 207 Pi0.5 RoboLab manipulation episodes (10 tasks, with semantic / spatial / scale language variations), produced in this repo's plan 007. It predicts future multi-view SD3 latents conditioned on robot actions, plus reward + critic heads.

Results (held-out validation, lower is better)

Training: batch 6, 8000 steps, single H100, ~1h45m. Full detail in memory/weaver-finetune.md and plans/007-demo-collection.md.

What's in the checkpoint

• checkpoint.pt — the fine-tuned FlowWM (32-layer spatio-temporal transformer, ~929M params) + reward + critic heads. EMA weights included. The frozen CLIP text-encoder keys are stripped (loaded from HF at build time).
• config.yaml — the WEAVER config the checkpoint was trained/evaluated with.
• 2-camera model: img_keys = [wrist_left, exterior_1_left] (run twice, right/left, for 3-view panels).

Not included (loaded at runtime, gated): the SD3 VAE (stabilityai/stable-diffusion-3-medium-diffusers) and the CLIP text encoder (openai/clip-vit-large-patch14). You must accept the SD3 license on HuggingFace and hf auth login.

Download

pip install -U huggingface_hub
# gated deps: accept the SD3 license once at hf.co/stabilityai/stable-diffusion-3-medium-diffusers
huggingface-cli login

# the fine-tuned checkpoint (~15 GB)
huggingface-cli download <HF_REPO> --local-dir ./weaver-robolab-ft
# -> ./weaver-robolab-ft/{checkpoint.pt, config.yaml, README.md}

<HF_REPO> is printed by the uploader and recorded in memory/weaver-finetune.md after the push.

Use it

Clone upstream WEAVER (github.com/arnavkj1995/WEAVER) and install it (this repo's scripts/vm-install-weaver.sh automates the GCP/H100 setup, incl. third_party/{openpi,robometer}). Then point the eval/rollout entrypoints at the downloaded dir:

# zero-shot / open-loop rollouts + saved views (per-camera gt/pred .npy)
python -m weaver.generate_views --checkpoint ./weaver-robolab-ft --output-dir out --split val \
    --num-videos 4 --start-idx 0 --overrides dataset.path=<your_dataset> \
    dataset.norm_stats_path=<your_dataset>/norm_stats_relabel.json dataset.annotation_dir=annotation_rewards

This repo's scripts work directly with the downloaded checkpoint:

• Fine-tune further / on your own data: scripts/finetune-weaver-singlegpu.sh (env knobs: DATASET, PRETRAINED, EXP, STEPS, BATCH, COMPILE, VALID_FREQ, VIDEO_FREQ, LR, NWORKERS) or scripts/finetune-weaver-max.sh (probe max batch + compile + eval). Set PRETRAINED=./weaver-robolab-ft.
• Build a fine-tune dataset from RoboLab demos: scripts/make_weaver_finetune_dataset.py (SD3+CLIP encode, --task-instructions, --id-offset/--append for streaming).
• Prediction metrics / panels: scripts/weaver_pred_metrics.py, scripts/compose_base_vs_ft.py, scripts/weaver_rollout_modes.py (open-loop vs closed-loop).
• Policy-in-the-dream (Pi0.5 ↔ WM): scripts/weaver_dream_policy.py.

License & attribution

Derived from WEAVER (arnavkj1995 / NVLabs) — respect the upstream model license. The checkpoint contains only fine-tuned FlowWM + reward/critic weights; the SD3 VAE and CLIP encoder are not redistributed here and must be obtained from their gated sources under their own licenses.