qwen3vl-aokvqa

qwen3vl-aokvqa

LoRA adapter for Qwen/Qwen3-VL-2B-Instruct, trained with the VLM2Vec contrastive-embedding recipe on the A-OKVQA subset of MMEB-train only (single V100 32GB, batch size 64, 440 steps, ~1.6 epochs).

This turns the backbone into a multimodal embedding model (last-token pooled, L2-normalized), not a generative chat model — load it with VLM2Vec's MMEBModel.load() (src/model.py), not AutoModelForCausalLM.

Training config

• Base model: Qwen/Qwen3-VL-2B-Instruct
• LoRA: r=8, alpha=16, dropout=0.0, targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
• Data: MMEB-train A-OKVQA subset only
• Steps: 440

Eval (MMEB-eval A-OKVQA test, full ~894-candidate pool, paper-faithful Precision@1)

• P@1 = 0.210 (210/1000), vs. chance ≈ 0.0011
• For reference, the VLM2Vec paper (LLaVA-1.6-7B, full 20-dataset joint training) reports 49.9% P@1 on the VQA meta-category. This checkpoint is trained on a single VQA dataset with far less compute/backbone capacity, so the gap is expected — it is not a reproduction of that number.

Usage

Requires the VLM2Vec repo (branch with Qwen3-VL + Volta/sdpa support).

from src.arguments import ModelArguments
from src.model import MMEBModel

model_args = ModelArguments(
    model_name="Qwen/Qwen3-VL-2B-Instruct",
    model_backbone="qwen3_vl",
    checkpoint_path="o-taco/qwen3vl-aokvqa",
    pooling="last",
    normalize=True,
    lora=True,
)
model = MMEBModel.load(model_args, is_trainable=False)

Framework versions

• PEFT 0.11.1