Downloads · 30 days
5
21% of all-time downloads
gavinzsmeng/visfineval-qwen3vl-8b-lora-balanced
visfineval-qwen3vl-8b-lora-balanced is a machine learning model from gavinzsmeng. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter for Qwen3-VL-8B-Instruct, fine-tuned on VisFinEval with the training set class-balanced by resampling: all six (question-type × label) combinations were upsampled to 6,725 examples each — 40,350 rows tota…
Downloads · 30 days
5
21% of all-time downloads
All-time downloads
24
Public
Repo size
175 MB
Likes
0
Public
Click a slice to open those files.
.safetensors175 MB · 100%
From the Hugging Face model README
LoRA adapter for Qwen3-VL-8B-Instruct, fine-tuned on VisFinEval with the training set class-balanced by resampling: all six (question-type × label) combinations were upsampled to 6,725 examples each — 40,350 rows total, versus 13,465 for the natural split.
📄 Full audit, code and reproduction steps: gavinzsmeng/visfineval-reliability
This is the ablation arm. The base model collapses onto the majority class on True/False questions ("No" recall 39.7%, below chance). The obvious hypothesis is imbalanced training data. This adapter tests it.
The hypothesis is wrong. Class balancing gave no benefit.
Same held-out test split (2,889 questions, report-level split, zero overlap), same prompt, greedy decoding.
| Condition | MC raw | TF raw | TF macro-recall | Recall @ "No" (95% Wilson CI) |
|---|---|---|---|---|
| Qwen3-VL-8B (base) | 73.70% | 75.23% | 63.73% | 39.7% [31.2, 48.8] |
| LoRA natural (sibling) | 84.13% | 84.01% | 79.15% | 69.0% [60.1, 76.7] |
| this adapter (balanced) | 82.78% | 84.01% | 79.43% | 69.8% [60.9, 77.4] |
Paired per-sample comparison against the natural adapter:
natural -> balanced fixed 84 broke 117 net -33 McNemar p = 0.024
The minority-class collapse was not caused by imbalanced training data. Standard SFT on the natural distribution already fixes it — the improvement appears to come from the model learning the task (reading the chart) rather than from seeing more minority examples. That mechanism is a hypothesis, not something this experiment proves.
The practical takeaway: before resampling a skewed dataset, check whether plain SFT already solves the problem. Here it did.
Identical to the natural adapter except for the data:
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Method | LoRA, r=16, α=32, target_modules=all-linear |
| Frozen | Vision encoder and aligner (freeze_vit=true) |
| Trainable params | 43.6M (0.50% of 8.81B) |
| Data | 40,350 rows (6 × 6,725), from the same 2,322 training reports |
| Epochs | 2 (1,262 steps) |
| Effective batch | 1 × 16 grad-accum × 4 GPUs |
| LR | 1e-4, cosine, 5% warmup |
| Precision | bfloat16 |
| Framework | ms-swift 4.6.0.dev0 |
| Hardware | 4× RTX 4090, DDP, ~4h55m |
Identical to the sibling adapter — see
visfineval-qwen3vl-8b-lora-natural
for the full snippet and the exact prompts.
⚠ Use the documented prompt verbatim. Prompt wording moves minority-class recall by up to 23 points on this task; a different phrasing will not reproduce these numbers.
Same as the natural adapter, plus:
Training images come from Chinese broker research reports and are not redistributed.
Obtain VisFinEval from
SUFE-AIFLM-Lab/VisFinEval.
Derivative of Qwen/Qwen3-VL-8B-Instruct (Apache-2.0), released under Apache-2.0.
@misc{meng2026visfinevalrel,
title = {VisFinEval Reliability Audit: what reported accuracy hides},
author = {Meng, Gavin C.},
year = {2026},
url = {https://github.com/gavinzsmeng/visfineval-reliability}
}