multipano-13clip-fillin

STATUS 2026-09-05: DONE. All 78 result JSONs are uploaded under results/. Jump to RESULTS UPLOADED for where they are, how to merge them, and what had to change to run this package on another machine. The instructions below are the original ones.

multipano-13clip-fillin

Goal: compute camera_da3.json and adherence_3d.json for 13 synth clips that are missing them, across 3 multipano models (Model_ours_multipano_iter4k_v2, iter5k_v2, iter6k_v2), then hand the results back so they can be merged into the main eval server's EVAL_Results_oldFINAL / EVAL_FINAL.

Why they're missing: these 3 models' point-prompt eval originally ran through a GT-path resolver (benchmark_object_masking.remap_path()) that redirected to a now-retired BENCHMARK/NEW curation snapshot missing these 13 clips, so camera_da3/3d_adherence silently skipped them (no crash, no file written). That redirect has since been removed in the code included here — GT now resolves directly against BENCHMARK/FINAL, where all 13 clips exist. iter7k_v2 (evaluated after the fix) already has full coverage and needs nothing from this package.

This repo is private and self-contained: code, the exact 13×3 model outputs, the 13 clips' GT videos, and the GPU environment (SAM3 + DA3 checkpoints + a matching Python venv). No other access to the original server is required.

---

0. Prerequisites on this (new) machine

• An NVIDIA GPU with CUDA available (nvidia-smi works, torch.cuda.is_available() is True).
• ~30GB free disk (venv ~9.2G, SAM3 ckpt ~6.5G, DA3 ckpt ~6.8G, clip data ~1.9G).
• huggingface_hub's CLI (pip install -U huggingface_hub if hf isn't already available) and a way to read the HF token (ask the user for it directly — do not ask them to paste it in chat; have them run export HF_TOKEN=hf_... in their own shell, or point you at wherever they've stored it, e.g. an env file you can source without printing it).

1. Download this repo

export HF_TOKEN=...   # from the user, never echoed/printed
mkdir -p ~/multipano_fillin
hf download kimhosoo/multipano-13clip-fillin --repo-type model --local-dir ~/multipano_fillin

If this times out or hangs on large files, retry with xet disabled (the upload side hit the same issue and this fixed it):

HF_HUB_DISABLE_XET=1 hf download kimhosoo/multipano-13clip-fillin --repo-type model --local-dir ~/multipano_fillin

After download, ~/multipano_fillin/ contains:

environment/sam3-env.tar                  <- the sam3-env venv, packaged as ONE tar file (~9.9G)
code/NHNHOME/hyunwook/khs/codes/...        <- eval scripts + their deps
data/NHNHOME/hyunwook/khs/BENCH_OUTPUT/... <- the 13×3 clips needing eval
data/NHNHOME/hyunwook/khs/JointGen-backup/BENCHMARK/FINAL/...  <- GT for those 13 clips
checkpoints/hf_cache/...                   <- SAM3 + DA3 weights (HF hub cache layout)
checkpoints/torch_cache/...                <- AlexNet weights (LPIPS backbone, torch hub cache layout)

environment/sam3-env.tar is a single tar file on purpose — the venv has ~52,000 files, and uploading/downloading it as loose files previously blew through HF's per-hour commit rate limit. Extract it once:

cd ~/multipano_fillin/environment
tar -xf sam3-env.tar   # produces ~/multipano_fillin/environment/sam3-env/

2. Put code + data at the exact absolute paths the scripts expect

The eval scripts have a few hardcoded absolute paths (/NHNHOME/hyunwook/khs/...). Recreate that prefix locally (you almost certainly have root on a fresh GPU box):

sudo mkdir -p /NHNHOME/hyunwook/khs
sudo chown -R "$(whoami)" /NHNHOME

rsync -a ~/multipano_fillin/code/NHNHOME/ /NHNHOME/
rsync -a ~/multipano_fillin/data/NHNHOME/ /NHNHOME/

(rsync -a merges the two NHNHOME/hyunwook/khs/... trees correctly — code/ supplies codes/ + JointGen-backup/Projects/sam3/{sam3,sam3.egg-info,scripts/track_and_extract_masks.py}, data/ supplies BENCH_OUTPUT/ + JointGen-backup/BENCHMARK/FINAL/... — no overwrite conflicts between them.)

You should now have exactly:

• /NHNHOME/hyunwook/khs/codes/{benchmark_object_masking.py,gt_model_transforms.py,panorama_to_perspective_crop.py,metric/eval_camera_da3.py,metric/eval_3d_adherence.py}
• /NHNHOME/hyunwook/khs/JointGen-backup/Projects/sam3/{sam3/,sam3.egg-info/,scripts/track_and_extract_masks.py}
• /NHNHOME/hyunwook/khs/BENCH_OUTPUT/Model_ours_multipano_iter{4k,5k,6k}_v2__missingsynth/Syn/<13 clip dirs>/{output.mp4,metadata.json}
• /NHNHOME/hyunwook/khs/JointGen-backup/BENCHMARK/FINAL/synth/perspective/<Town>/<clip>/<traj>.gt.mp4 (13 files)

3. Verify the venv

~/multipano_fillin/environment/sam3-env/bin/python3 -c "import torch, cv2, lpips, depth_anything_3; from sam3.model_builder import build_sam3_video_predictor; print('torch', torch.__version__, 'cuda', torch.cuda.is_available())"

This should print cuda True. If it fails to even start (e.g. can't find the base interpreter), it's because the venv was created via uv referencing home = /usr/bin (i.e. it expects a system Python 3.12 at /usr/bin/python3 on this machine too) — check cat ~/multipano_fillin/environment/sam3-env/pyvenv.cfg. If your box's system Python doesn't match, stop and report back rather than trying to patch the venv; we'll figure out a fallback (rebuilding a fresh venv against the sam3 source now sitting at /NHNHOME/hyunwook/khs/JointGen-backup/Projects/sam3/sam3) together.

4. Point the checkpoints at the bundled cache (fully offline, no HF auth needed for the gated SAM3/DA3 repos)

export HF_HOME=~/multipano_fillin/checkpoints/hf_cache
export TORCH_HOME=~/multipano_fillin/checkpoints/torch_cache
export HF_HUB_OFFLINE=1

5. Run the eval — for each of the 3 models

cd /NHNHOME/hyunwook/khs/codes/metric
PY=~/multipano_fillin/environment/sam3-env/bin/python3

for MODEL in Model_ours_multipano_iter4k_v2__missingsynth \
             Model_ours_multipano_iter5k_v2__missingsynth \
             Model_ours_multipano_iter6k_v2__missingsynth; do
  DIR="/NHNHOME/hyunwook/khs/BENCH_OUTPUT/$MODEL"
  "$PY" eval_camera_da3.py   --model_output_dir "$DIR" --gpus 0 --workers 2 --stride 1
  "$PY" eval_3d_adherence.py --model_output_dir "$DIR" --gpus 0 --workers 2
done

Each clip folder should end up with a camera_da3.json and an adherence_3d.json written into it (13 clips × 3 models = 39 of each, 78 files total). Sanity-check before moving on:

for MODEL in Model_ours_multipano_iter4k_v2__missingsynth \
             Model_ours_multipano_iter5k_v2__missingsynth \
             Model_ours_multipano_iter6k_v2__missingsynth; do
  echo "$MODEL: $(find /NHNHOME/hyunwook/khs/BENCH_OUTPUT/$MODEL -name camera_da3.json | wc -l) camera_da3.json, \
$(find /NHNHOME/hyunwook/khs/BENCH_OUTPUT/$MODEL -name adherence_3d.json | wc -l) adherence_3d.json"
done

Expect 13 camera_da3.json, 13 adherence_3d.json on every line. If any clip is missing its file, re-run that model's command with --overwrite — a prior crash on one clip shouldn't silently corrupt the others, but check the script's own stdout for a FAILED line naming the clip.

The 13 clip IDs (same set for all 3 models) are:

synth__Town01__clip_0040__out_back_left_first
synth__Town01__clip_0047__out_back_right_first
synth__Town02__clip_0101__out_back_left_first
synth__Town02__clip_0189__out_back_left_first
synth__Town03__clip_0116__out_back_right_first
synth__Town03__clip_0141__out_back_right_first
synth__Town03__clip_0158__out_back_right_first
synth__Town05__clip_0128__out_back_right_first
synth__Town05__clip_0132__out_back_left_first
synth__Town05__clip_0134__out_back_left_first
synth__Town05__clip_0254__out_back_right_first
synth__Town10HD_Opt__clip_0025__out_back_left_first
synth__Town10HD_Opt__clip_0149__out_back_right_first

6. Upload just the results back to this same repo

Don't re-upload output.mp4 (already in the repo, large, unchanged) — only the two new JSON files per clip:

export HF_HOME=~/multipano_fillin/checkpoints/hf_cache   # unset offline mode for this part
unset HF_HUB_OFFLINE

for MODEL in Model_ours_multipano_iter4k_v2__missingsynth \
             Model_ours_multipano_iter5k_v2__missingsynth \
             Model_ours_multipano_iter6k_v2__missingsynth; do
  HF_HUB_DISABLE_XET=1 hf upload kimhosoo/multipano-13clip-fillin \
    "/NHNHOME/hyunwook/khs/BENCH_OUTPUT/$MODEL" "results/$MODEL" \
    --repo-type model --include "*/camera_da3.json" --include "*/adherence_3d.json"
done

(HF_HUB_DISABLE_XET=1 because plain uploads hit an httpx.ReadTimeout on this repo without it — unclear why, but disabling xet fixed every retry during setup. If uploads still hang, try again once or twice before reporting back; it was intermittent, not a hard failure.)

7. Tell the user it's done

Once all three results/Model_ours_multipano_iter*_v2__missingsynth/ prefixes are uploaded (verify with hf repo-files kimhosoo/multipano-13clip-fillin or just re-check via the huggingface_hub API), report back to the user that the 13-clip fill-in is uploaded and ready to be pulled down on the original server. You don't need to do anything else — merging these results into EVAL_Results_oldFINAL / EVAL_FINAL happens back on that server.

---

Reference: why only these 2 metrics, and only these 3 models

• fid/fvd for these 13 clips were not affected the same way (checked separately) — this package is scoped to exactly what's missing: camera_da3 + 3d_adherence, for iter4k_v2/iter5k_v2/iter6k_v2 only.
• outer_view/object_consistency/oov_direction_* (the bbox-prompt metrics) already had full 100/100 coverage for all these models the whole time — a different code path (final_benchmark/benchmark_object_masking_{bboxprompt,pointprompt}.py) already pinned GT resolution to BENCHMARK/FINAL independently, before this fix existed. Nothing to do there.

---

✅ RESULTS UPLOADED (2026-09-05) — read this section first

The fill-in was run to completion on a second GPU box (8×H100, no shared filesystem with the original server). 13 clips × 3 models × 2 metrics = 78 JSON files are now in this repo under results/:

results/Model_ours_multipano_iter{4k,5k,6k}_v2__missingsynth/Syn/<sample_id>/camera_da3.json
results/Model_ours_multipano_iter{4k,5k,6k}_v2__missingsynth/Syn/<sample_id>/adherence_3d.json

<sample_id> matches BENCH_OUTPUT/<MODEL>/Syn/<sample_id> exactly, so merging back is a straight copy into each clip folder.

hf download kimhosoo/multipano-13clip-fillin --repo-type model \
  --include "results/*" --local-dir /tmp/fillin_results
# then, per model, copy results/<MODEL>/Syn/<clip>/*.json into that clip's folder

Every clip succeeded — 0 FAILED lines, both batch drivers exited 0, coverage check reported 13 camera_da3.json, 13 adherence_3d.json for all three models.

Aggregate over the 13 clips (also in each model's own EVAL_Results/{camera_da3,3d_adherence}/<MODEL>.json, not uploaded):

3d_adherence ran with dynamic-object masking on (use_masks: true), 77 frames per clip; camera_da3 ran at --stride 1 (all 77 frames).

What had to be changed to run this package elsewhere

1. A missing input: caption_selected/*.gt.meta.json (fixed mid-run)

eval_3d_adherence.py::load_clip_info() reads metadata.json's caption_path to get prompts (the SAM3 category prompts, e.g. ["pedestrian","vehicle"]). The first upload of this repo shipped only the 13 .gt.mp4 files, so every clip returned no_data and not one adherence_3d.json would have been written — silently, exactly like the bug this package exists to fix. The 13 caption_selected/<traj>.gt.meta.json files were uploaded afterwards and are now part of the repo; anyone re-running this needs them. (w2cs_path is also unresolvable in this package, but neither metric reads it, so that is fine.)

2. The hardcoded /NHNHOME/hyunwook/khs prefix was removed

Step 2 of the original instructions (create /NHNHOME at the filesystem root) was not acceptable on the second box, so the five scripts now derive their root from their own location, with an env override:

KHS_ROOT = os.environ.get("KHS_ROOT", <dirname walk-up from __file__>)

GT_PATH_PREFIX_OLD = "/home/work/.local/khs" was left alone on purpose — that is the prefix baked into every clip's metadata.json, which remap_path() rewrites onto GT_PATH_PREFIX_NEW.

Two incidental fixes while doing this:

• panorama_to_perspective_crop.py had no import os at all (it only ever used pathlib); referencing os there would have been a NameError.
• The /NHNHOME/hyunwook/khs strings inside docstrings and --help text were rewritten to $KHS_ROOT so nobody copy-pastes a dead path.

Caches (codes/gt_transform_cache/, codes/metric/gt_mask_cache/) were already __file__-relative and needed no change.

3. The venv's editable sam3 install pointed at /NHNHOME

environment/sam3-env.tar contains an editable (pip install -e) install of sam3, so after extracting, two files still pointed at the old absolute path and import sam3 failed:

lib/python3.12/site-packages/__editable___sam3_0_1_0_finder.py
lib/python3.12/site-packages/sam3-0.1.0.dist-info/direct_url.json

Rewriting /NHNHOME/hyunwook/khs → the local root in those two files was enough. pyvenv.cfg says home = /usr/bin + version_info = 3.12.3, which matched this box's system Python, so no rebuild was needed. Smoke test after the fix: torch 2.10.0+cu128, cuda True, 8 GPUs, sam3 resolving to the local tree.

4. GPU sizing — DA3 needs a whole GPU per clip

eval_camera_da3.py at --stride 1 peaks at ~40–47 GB per clip (DA3-GIANT over 77 frames at 1280×704, twice: GT and pred). Two workers per 80 GB H100 OOMs (CUDA out of memory … Process X has 47.35 GiB in use). What worked:

Both pipelines ran in parallel on disjoint GPUs; within a pipeline the three models must stay sequential — 3d_adherence shares one GT SAM3 mask cache (gt_mask_cache/) across models, so concurrent models would both duplicate the GT SAM3 pass and race on the same .npz path. Wall clock for all 78 results: 28 minutes.

5. Downloading environment/sam3-env.tar (9.9 GB) needed a workaround

The other 30 GB of this repo pulled at ~70 MB/s, but this one file crawled at ~0.2 MB/s on a single connection (hf download, xet on or off) and would have taken ~10 h. Fetching it as parallel HTTP range requests fixed it — 48 chunks, then 16 MB sub-chunks over 64 threads, curl -r <start>-<end> -C - against https://huggingface.co/<repo>/resolve/main/environment/sam3-env.tar with the HF token as a bearer header, concatenated in order (verified byte count 9941821440, tar -tf clean). Throughput scaled roughly with connection count, so this looks like per-connection shaping on that object, not a network limit.