URIS-Qwen2.5-VL-7B-RefCOCO-LoRA

URIS — Qwen2.5-VL-7B RefCOCO Grounding (DoRA/LoRA+)

DoRA adapter fine-tuning Qwen/Qwen2.5-VL-7B-Instruct for referring-expression comprehension (visual reference resolution) on RefCOCO. Given a referring expression, the model predicts a bounding box in [0,1000] normalized integer coordinates.

Training

• Base: Qwen/Qwen2.5-VL-7B-Instruct
• Method: DoRA (r=32, alpha=64, dropout=0.05) on attn+MLP proj; merger/projector full fine-tuned; vision tower frozen
• Data: RefCOCO (lmms-lab/refcoco), 50k ShareGPT-format samples (see data/)
• Schedule: 2 epochs, lr=1e-5 cosine, effective batch 96, 1042 steps
• Final train loss: 0.865
• Hardware: 6x Ascend 910B2 (64GB), torch_npu 2.10, transformers 4.57.6, peft 0.19.1

Evaluation (RefCOCO val, 500 queries)

(*) base parsed fairly from its native bbox_2d JSON output and rescaled to [0,1000]. (**) base does not emit the target [0,1000] (x1,y1),(x2,y2) format; LoRA does, 100% of the time.

Temporal references (synthetic benchmark, 120 scenes / 360 queries)

Constructed object-presence-history benchmark testing temporal referring expressions ("the object that was there earlier"):

The 66-point gap shows temporal references require an explicit memory module. This benchmark is synthetic — it demonstrates the mechanism, not natural-video performance.

Detection latency (YOLOv8n)

• ~8 ms median on NVIDIA H20 GPU
• ~305 ms median on CPU (Ascend training server has no GPU)

Data

• data/refcoco_en_sft.jsonl — 50k training samples (image filename + referring expr + bbox)
• data/merged_refs.jsonl — merged RefCOCO refs
• Images: COCO train2014 (public). Each sample references a COCO filename.

Usage

from transformers import AutoModelForImageTextToText
from peft import PeftModel
base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct", dtype="bfloat16")
model = PeftModel.from_pretrained(
    base, "Steven668866/URIS-Qwen2.5-VL-7B-RefCOCO-LoRA")