3d-ckpts-v3

3d-ckpts-v3 — Thinking-in-Space GRPO checkpoints (Qwen2.5-VL-3B)

Qwen2.5-VL-3B-Instruct fine-tuned with GRPO (verl) to answer ScanQA-style 3D spatial questions by actively exploring ScanNet scenes: the policy calls a render_view_tool (camera teleport via coordinates), inspects the returned views, then answers. Reward: LLM-judge accuracy (Qwen3-30B, CORRECT/INCORRECT) + format + small per-call tool bonus + geometric grounding (final view must contain the answer object's 3D box, occlusion-checked raycast). Max 6 assistant turns. No raw ScanNet data is included here.

Checkpoints & performance

• Official val = eval_official_val_v2 (n=256, held-out ScanQA-style, pure judge accuracy, no bonuses). Historical bar to beat: 0.176 (best previous healthy run) — v9.1c-s120 clears it at 0.195.
• Tiny val (n=64) is the during-training monitor; noisy, use for relative comparison only.
• All runs: 1 sample/question judge scoring, temperature 0.7 rollouts, KL 0.005, train set train_filtered_v9 (16,777 QA / 562 ScanNet scenes).

Usage

Standard HF Qwen2.5-VL loading; each folder is a complete model.

from transformers import AutoModelForVision2Seq, AutoProcessor
m = AutoModelForVision2Seq.from_pretrained("Icey444/3d-ckpts-v3", subfolder="v9.1c-s120")
p = AutoProcessor.from_pretrained("Icey444/3d-ckpts-v3", subfolder="v9.1c-s120")

The interactive 3D behavior requires the render-tool harness from the training repo (multi-turn tool loop over ScanNet meshes); as plain VLMs these behave like Qwen2.5-VL with improved spatial question answering.