qwen25vl-3b-perdataset-derivatives

Qwen2.5-VL-3B — Per-Dataset Fine-Tuning Derivatives (LoRA + Full FT)

10 fine-tuned derivatives of Qwen/Qwen2.5-VL-3B-Instruct, one LoRA and one full fine-tune for each of 5 vision-language datasets. Language-model backbone only — the vision tower and multimodal projector are frozen in both variants.

⚠️ Research artifacts: each model is trained on a 3,000-example slice of one dataset for 3 epochs.

Shared training recipe (all 10)

• Base: Qwen/Qwen2.5-VL-3B-Instruct
• Frozen: vision tower + multimodal projector (freeze_vision_tower: true, freeze_multi_modal_projector: true) — only the LM backbone is trained.
• Data: max_samples = 3000 per dataset, 3 epochs, cutoff_len = 2048.
• Framework: LLaMA-Factory 0.9.3, transformers 4.52.4.
• LoRA variant: target q_proj, v_proj; rank 16, alpha 32; lr 2e-4.
• Full variant: DeepSpeed ZeRO-3; lr 1e-5; image_max_pixels = 401408; trainable = 100 % of the LM (vision/projector still frozen).

Contents

Datasets: MathV360K, PaintingForm, ST-VQA, TextVQA, Visual7W.

Usage

LoRA (base + PEFT):

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
base = Qwen2_5_VLForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "DSA1/qwen25vl-3b-perdataset-derivatives", subfolder="lora/textvqa")
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

Full FT (self-contained LM weights; processor/chat-template from base):

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
model = Qwen2_5_VLForConditionalGeneration.from_pretrained("DSA1/qwen25vl-3b-perdataset-derivatives", subfolder="full/textvqa", torch_dtype="auto", device_map="auto")
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

Notes

• train_loss = mean training loss over all optimizer steps (1014 steps per model at global batch 8 for LoRA; 129 steps at global batch 64 for full).
• Full-FT models ship LM weights only; the vision tower + projector are byte-identical to the base — load processor/chat-template from Qwen/Qwen2.5-VL-3B-Instruct.
• A 7B counterpart of these derivatives exists separately.

License

Inherits the base model's license (Qwen/Qwen2.5-VL-3B-Instruct). Datasets retain their respective licenses.