Downloads · 30 days
2
6% of all-time downloads
Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-v5-lora
qwen3vl-4b-wiki-screenshot-multik-3x-v5-lora is a image-text-to-text model from Chrisyichuan. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
Snapshot repo. This is a frozen snapshot of the v5 recipe at checkpoint-16000. The main multik-3x-lora repo tracks the current best 3x adapter (currently identical to this one). Sister snapshot for the prior recipe (1…
Downloads · 30 days
2
6% of all-time downloads
All-time downloads
33
Public
Repo size
1.3 GB
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB · 99%
From the Hugging Face model README
Snapshot repo. This is a frozen snapshot of the v5 recipe at checkpoint-16000. The main
multik-3x-lorarepo tracks the current best 3x adapter (currently identical to this one). Sister snapshot for the prior recipe (1:1 mixed) lives atChrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-v3-loraif archived.
LoRA adapter for Qwen/Qwen3-VL-4B-Instruct, fine-tuned as the reader in a retrieval-augmented Wikipedia-screenshot QA pipeline.
This adapter reads any number of screenshots from 1 to 6 at inference time, at a fixed 3x pixel compression per image. It exceeds the uncompressed no-SFT baseline at k≥2 and matches it at k=1 within 0.054.
| k (images per sample) | base@0x (no comp, no SFT) | v3 (1:1 mixed) | v5 (2:1 top3:vark) this | Δ vs v3 | Δ vs base@0x |
|---|---|---|---|---|---|
| 1 | 0.958 | 0.884 | 0.904 | +0.020 | -0.054 |
| 2 | 0.912 | 0.878 | 0.918 | +0.040 | +0.006 |
| 3 | 0.892 | 0.892 | 0.932 | +0.040 | +0.040 |
| 4 | 0.856 | 0.862 | 0.884 | +0.022 | +0.028 |
| avg | 0.905 | 0.879 | 0.910 | +0.031 | +0.005 |
Compared with v3 (1:1 mixed):
| Field | v3 (prior) | v5 (this) |
|---|---|---|
| Datasets | multimage_top3_train + multimage_vark_train (1:1) | multimage_top3_train + multimage_top3_train_dup + multimage_vark_train (2:1 top3:vark) |
| Samples per epoch | 208k | 312k (+50%) |
| LoRA rank | 256 | 256 (unchanged) |
| LoRA alpha | 256 | 256 (unchanged) |
| LoRA targets | all linear layers (incl. ViT) | all linear layers (incl. ViT) |
freeze_vision_tower | false | false |
| Optimizer | AdamW, cosine, peak 1e-5, warmup 3% | identical |
| Effective batch | 32 (1 × 8 GPUs × 4 grad_accum) | identical |
cutoff_len | 8192 | 8192 |
| Epochs | 2 (final adapter, step 13004) | 2 trained (full 19506 steps), but released checkpoint is step 16000 (~1.64 epochs) — peak eval exact-match before mild overfit |
| Hardware | 8× H100 80GB, DeepSpeed ZeRO-2, bf16 | identical |
| Wallclock | ~7h | ~9h |
In retrieval-augmented Wikipedia-screenshot QA, k=3 is the most common deployment shape (most retrievers return the top 3 docs). The variable-k (vark) split alone undertrained the model on this exact shape because its samples are uniform over k ∈ {1..6} (~17k per k). Adding a duplicate of the fixed-k=3 (top3) split shifts ~67% of training samples to k=3, which:
| step | epoch | eval_loss | eval_em | LLM-judge avg (k=1..4) |
|---|---|---|---|---|
| 14400 | 1.48 | 0.163 | 0.766 | (not benchmarked) |
| 16000 | 1.64 | 0.164 | 0.772 | 0.910 ← released |
| 17600 | 1.80 | 0.165 | 0.769 | (not benchmarked) |
| 19200 | 1.97 | 0.165 | 0.769 | (not benchmarked) |
| 19506 (final) | 2.00 | 0.166 | 0.768 | 0.901 (this trails ckpt-16000 at k=1/2/3, ties at k=4) |
Train loss in epoch 2 was 0.04-0.06 (vs epoch 1's 0.10-0.20) — the model continued to fit the training distribution well, but the gains stopped transferring to held-out test around step 16000. Pull a mid-epoch-2 checkpoint at peak eval em.
Built from the Chrisyichuan/screenshot-training-natural-filtered-v2 QA dataset (~104k train examples):
dataset: to oversample 2×.1/sqrt(3) via PIL LANCZOS).<image>×k \n {query} → gold answer.Effective per-epoch distribution:
Total per-epoch samples: 312k (1.5× v3's 208k). Train wallclock scales accordingly.
Gold-retrieval rate at top-6 across splits: ~75%. When gold is missing from retrieval, gold is still always included by construction in both splits.
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
import torch
base = "Qwen/Qwen3-VL-4B-Instruct"
adapter = "Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-v5-lora" # or "Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-lora" for the same weights via main repo
model = Qwen3VLForConditionalGeneration.from_pretrained(base, torch_dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(model, adapter).merge_and_unload()
processor = AutoProcessor.from_pretrained(base)
# k can be any integer in 1..6. Images must already be 3x-compressed (each dim × 1/sqrt(3)).
messages = [{"role": "user", "content": [
{"type": "image", "image": img_1},
# ... up to img_6 ...
{"type": "text", "text": your_question},
]}]
# ... standard Qwen3-VL inference
Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-2x-lora, Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-4x-lora. Latest 3x: Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-lora.