Qwen3-VL-8B-Instruct-scireason

top-papers/Qwen3-VL-8B-Instruct-scireason

This repository contains the fine-tuned SciReason VLM artifacts produced by the DataSphere SFT + GRPO pipeline.

Contents

• Root files: final GRPO adapter and processor files copied from outputs/hf_top_papers_qwen3vl_8b_grpo_lora for convenient loading.
• artifacts/sft_lora/: SFT LoRA adapter directory copied from outputs/hf_top_papers_qwen3vl_8b_sft_lora.
• artifacts/grpo_lora/: complete final GRPO output directory copied from outputs/hf_top_papers_qwen3vl_8b_grpo_lora.
• artifacts/archives/: compressed .tar.gz archives produced by the job.
• artifacts/data/: generated train/eval JSONL files and dataset summary.
• artifacts/reports/: budget, final summary, upload manifest and runtime reports.

Training metadata

• Base model: Qwen/Qwen3-VL-8B-Instruct
• Dataset: top-papers/top-papers-graph-experts-data
• Output prefix: hf_top_papers_qwen3vl_8b
• Uploaded at UTC: 2026-06-22T07:38:23Z

Loading note

The root of this repository is prepared as the final GRPO adapter directory. For LoRA/PEFT loading, use the same base model listed above and load this repository as the adapter. The complete SFT and GRPO directories are also preserved under artifacts/ for auditability and reproducibility.

---

Evaluation: base vs SFT-LoRA vs SFT+GRPO-LoRA

Technical report for the A/B evaluation of top-papers/Qwen3-VL-8B-Instruct-scireason against stock Qwen/Qwen3-VL-8B-Instruct, run as six Kaggle notebooks (T4 x2, 4-bit NF4).

1. TL;DR

Fine-tuning works where it was trained and degrades elsewhere:

• +0.2073 F1 on in-domain structured extraction (0.2715 -> 0.4788) - the largest and most trustworthy effect.
• +0.0440 on MathVista - a modest, real transfer gain.
• -0.0356 MMMU, -0.0133 HallusionBench, -0.0133 POPE - mild but consistent erosion.
• -0.7462 OCRBench (0.8625 -> 0.1163) - catastrophic, and the single most important finding in this report. It contradicts the "no general-ability degradation" hypothesis that the nb5 control contour was designed to test.

Two of the six notebooks produced no usable data at all (nb2, nb6), one is partial (nb5), and one headline metric (indomain_grpo) was broken in the scorer. Those gaps are documented in section 4 rather than hidden.

2. Headline results

Scores as reported by the run pipeline. n is the number of scored items.

2.1 In-domain: the core win

On indomain_sft (255 Russian-language expert extraction tasks) the adapter nearly doubles F1. Both fine-tuned configs are essentially tied (0.4775 vs 0.4788), so GRPO adds almost nothing beyond SFT on this benchmark - the gain comes from SFT.

em (exact match) rises 0.0000 -> 0.0353/0.0392, so the gain is not merely partial-credit noise on free-form text.

Note the base model already emits valid JSON for 99.6% of items but with semantically wrong content (e.g. "subject": "Bachelier" where gold expects a DOI). SFT changes the content and the schema discipline, not the output format.

3. The OCRBench collapse (most important finding)

OCRBench is the general-OCR control contour. Comparing the same 215 questions answered by both configs (paired, no sampling confound):

169 correct answers lost, 0 gained. The regression is not marginal - it is a near-total failure of general OCR ability.

The failure mode is characteristic: the SFT model emits plausible, well-formed but semantically wrong text, degrading across every one of the 25 OCRBench subsets (ChartQA 0.950->0.000, docVQA 0.950->0.000, POIE 0.963->0.000, SROIE 0.815->0.000, HME100k 0.700->0.000, ...). Examples:

This is the signature of catastrophic forgetting / output-distribution collapse: the adapter drives generation toward the narrow in-domain response style at the expense of faithful transcription.

4. Data-quality issues found while auditing

This section matters as much as section 2: three of the headline numbers as originally produced were wrong or missing.

4.1 indomain_grpo 0.0 is a scorer bug, not a model result

All three configs scored exactly 0.0. That is impossible for two different adapters and a stock baseline, and it is an artifact:

• The models emit a verdict field from the vocabulary {supported, not_supported, unsupported}.
• The gold labels use a different vocabulary: {accepted, rejected, uncertain, needs_time_fix, needs_evidence_fix, added}.
• The scorer's parser collapses every prediction to the literal string "support", which matches no gold label, so accuracy is structurally 0 for all configs.

Re-scoring the identical raw predictions with a vocabulary map gives:

Two things follow. First, the real verdict accuracy is ~51% for base and grpo, not 0%. Second - and this is a genuine finding - SFT badly damages output-schema compliance: 105/196 SFT outputs use invented verdicts (leads_to x90, supports x46, unconfirmed, unresolved, exceeds, ...), versus 0/196 for base and grpo. GRPO repairs this back to a clean schema, which is a real, attributable benefit of the GRPO stage.

Caveat: the rescored numbers are this analysis's own reconstruction from raw predictions, not the pipeline's output. They should be treated as indicative.

4.2 nb2 (CharXiv + ChartQA) produced nothing

Both benchmarks materialized 0 samples from 2500 candidates: every image failed with unsupported image type bytes, and ChartQA's split lookup failed repeatedly ({'train','val'} instead of test). scores.json is {}.

4.3 nb5 (control contour) is incomplete

Only OCRBench produced data. MMStar and DocVQA have no results at all, and the OCRBench SFT arm was cut short at 215/400 items when the worker hit No space left on device (os error 28), after which the GRPO arm crashed on adapter loading (exit -11 / -6). The 215-vs-400 asymmetry is why section 3 uses a paired comparison.

4.4 nb6 (Russian probe) skipped by design

ru_probe.jsonl was never attached, so the notebook exited cleanly with empty results. This is a configuration gap, not a failure - but it means the Russian-transfer hypothesis is untested.

4.5 Other observed friction

• MMMU validation needed repeated retries against HTTP 503s before succeeding (n=900 OK).
• GPU OOM (CUDA out of memory, ~14.6 GiB T4) recurred during nb1 on one long sample.
• The final OCRBench SFT JSONL is truncated mid-record (1 unterminated line).

5. Interpretation

The pattern is coherent and the mechanism is legible:

1. In-domain SFT works. On the training distribution the adapter is clearly better, and the gain is large enough (+0.21 F1) to be real.
2. GRPO's value is behavioral, not accuracy. It adds ~nothing on in-domain F1 (+0.0013) and MathVista, but it restores schema compliance that SFT destroyed (105 -> 0 off-schema verdicts). That is a meaningful, measurable contribution.
3. General ability is genuinely damaged. MMMU, HallusionBench and POPE each lose a small but consistent amount; OCRBench loses almost everything. The "no catastrophic forgetting" claim the control contour was built to support is not supported by the data that was actually collected.

The most defensible summary: this is a specialist adapter that trades broad visual competence for in-domain precision. Whether that trade is acceptable depends on the deployment target - it is a poor choice for any general-purpose or OCR-heavy use, and a good one for narrow domain extraction.

6. Limitations

• Coverage gaps. ChartQA, CharXiv, MMStar, DocVQA and ru_probe were never measured. Every conclusion about "general ability" rests on OCRBench, MMMU, HallusionBench and POPE.
• No replication. Single run per config; no seed variance or confidence intervals. The -0.0133 deltas are within plausible noise; the +0.2073 and -0.7462 are not.
• Quantized inference. 4-bit NF4 for both base and adapters; absolute numbers are not comparable to published fp16 leaderboard figures.
• Unpaired where noted. Only OCRBench was paired item-by-item. The others compare aggregate scores, which is safe here because item counts match per benchmark.
• One metric is reconstructed. indomain_grpo is this report's own rescoring (4.1).
• Charts were validated structurally (valid PNG, correct dimensions) but not visually inspected in this session - the vision backend was unavailable.

7. Recommended follow-ups

1. Fix the verdict scorer (map supported/not_supported -> accepted/rejected) and re-run nb1. Do not report indomain_grpo as 0.0 anywhere.
2. Re-run nb2, nb5 (mmstar/docvqa) and nb6 with adequate disk headroom to close the coverage gaps - especially before making any broad capability claim.
3. Investigate the OCRBench collapse directly: is it prompt-format sensitivity, LoRA rank/scope, or training-data imbalance? Checking the adapter on plain OCR prompts with the in-domain prompt template suppressed would isolate the cause quickly.
4. Consider reducing SFT strength or mixing in general OCR data, since GRPO already fixes schema collapse and general-data mixing is the standard mitigation for forgetting.
5. Report per-subset metrics, which is what made the OCRBench failure legible here.

8. Reproducibility

Raw predictions, per-benchmark scores.json, summary.md, console logs and the analysis scripts are preserved in the evaluation workspace (results/):

• analysis.json - consolidated machine-readable results
• SUMMARY.md - condensed summary table
• scores_by_benchmark.png, deltas_grpo_vs_base.png, ocrbench_collapse.png
• outputs/ - raw *.jsonl predictions, per-stage scores.json, run.log, console.log
• notebooks_src/ - the six notebook sources as executed

Source notebooks: apollinariat/nb1-indomain, nb2-charts, nb3-science, nb4-verification, nb5-control, nb6-ru-probe.

---

Report generated from the Kaggle run outputs of 2026-08-01/02. All numbers are traceable to the raw per-item predictions retained alongside it.