Downloads · 30 days
68
50% of all-time downloads
gavinzsmeng/visfineval-qwen3vl-8b-lora-natural
visfineval-qwen3vl-8b-lora-natural is a machine learning model from gavinzsmeng. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter for Qwen3-VL-8B-Instruct, fine-tuned on VisFinEval (Chinese financial multimodal benchmark, EMNLP 2025) with the training data split by source research report to prevent leakage.
Downloads · 30 days
68
50% of all-time downloads
All-time downloads
137
Public
Repo size
175 MB
Likes
0
Public
Click a slice to open those files.
.safetensors175 MB · 100%
From the Hugging Face model README
LoRA adapter for Qwen3-VL-8B-Instruct, fine-tuned on VisFinEval (Chinese financial multimodal benchmark, EMNLP 2025) with the training data split by source research report to prevent leakage.
This is the natural distribution condition — the training set keeps its original label skew (58.5% of multiple-choice answers are "A"). No resampling.
📄 Full audit, code and reproduction steps: gavinzsmeng/visfineval-reliability
The base model collapses onto the majority class on True/False questions: its recall on the minority answer ("No") is 39.7% — below random chance (50%). We fine-tuned to see whether standard SFT fixes this, and measured it with per-sample paired tests rather than raw accuracy alone.
Evaluated on 2,889 held-out test questions (split by source report, zero overlap with training), same prompt as training, greedy decoding.
| Condition | MC raw | TF raw | TF macro-recall | Recall @ "No" (95% Wilson CI) |
|---|---|---|---|---|
| Qwen3-VL-8B (base) | 73.70% | 75.23% | 63.73% | 39.7% [31.2, 48.8] |
| + this adapter | 84.13% | 84.01% | 79.15% | 69.0% [60.1, 76.7] |
Paired per-sample comparison (identical global uid):
base -> this adapter fixed 430 broke 136 net +294 McNemar p < 0.001
The gain concentrates almost entirely on the minority class:
"Yes" recall (n=328) 87.8% -> 89.3% +1.5pp
"No" recall (n=116) 39.7% -> 69.0% +29.3pp (20x difference)
The confidence intervals for "No" recall ([31.2, 48.8] vs [60.1, 76.7]) do not overlap.
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Method | LoRA, r=16, α=32, target_modules=all-linear |
| Frozen | Vision encoder and aligner (freeze_vit=true) |
| Trainable params | 43.6M (0.50% of 8.81B) |
| Data | 13,465 questions / 2,322 reports (report-level split, 0 leakage) |
| Epochs | 2 (422 steps) |
| Effective batch | 1 × 16 grad-accum × 4 GPUs |
| LR | 1e-4, cosine, 5% warmup |
| Precision | bfloat16 |
| Framework | ms-swift 4.6.0.dev0 |
| Hardware | 4× RTX 4090, DDP, ~1h36m |
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
REPO = "gavinzsmeng/visfineval-qwen3vl-8b-lora-natural"
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
base = AutoModelForImageTextToText.from_pretrained(
"Qwen/Qwen3-VL-8B-Instruct", dtype="bfloat16", device_map="cuda"
)
model = PeftModel.from_pretrained(base, REPO).merge_and_unload().eval()
from PIL import Image
img = Image.open("chart.jpg").convert("RGB")
PROMPT = (
"请仔细阅读图片,回答问题,只输出正确选项的字母(A/B/C/D),不要解释。\n\n"
"问题:{q}\nA. {a}\nB. {b}\nC. {c}\nD. {d}\n\n答案:"
)
msgs = [{"role": "user", "content": [
{"type": "image", "image": img},
{"type": "text", "text": PROMPT.format(q="...", a="...", b="...", c="...", d="...")},
]}]
inputs = processor.apply_chat_template(msgs, add_generation_prompt=True,
tokenize=True, return_dict=True,
return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
⚠ Use the prompt above verbatim. Prompt wording moves minority-class recall by up to 23 points on this task (see Finding 4 in the audit), so a different phrasing will not reproduce these numbers.
For True/False questions, the prompt is:
请仔细阅读图片,回答问题,只回答「是」或「否」,不要解释。
问题:{question}
答案:
Training images come from Chinese broker research reports and are not redistributed
here. Obtain VisFinEval from its official channels:
SUFE-AIFLM-Lab/VisFinEval.
This adapter is a derivative of Qwen/Qwen3-VL-8B-Instruct (Apache-2.0) and is released
under Apache-2.0.
@misc{meng2026visfinevalrel,
title = {VisFinEval Reliability Audit: what reported accuracy hides},
author = {Meng, Gavin C.},
year = {2026},
url = {https://github.com/gavinzsmeng/visfineval-reliability}
}