PaperScorer-3B-Vision-arXiv

PaperScorer-3B-Vision-arXiv

SFT fine-tune of Qwen/Qwen2.5-VL-3B-Instruct trained to predict ICLR-style Accept/Reject verdicts on academic papers from the paper text plus per-page screenshots.

• Modality: text + per-page image screenshots
• Training data: arxiv-large (71k balanced_per_venue)
• Checkpoint: step 17724, end of epoch 4 of 4 (3B → ep4 by convention)
• Hyperparams: LR 1e-6, cosine_then_constant scheduler (decay_ratio 0.75, min_lr_rate 0.001), batch size 16, cutoff_len 24480, framework LLaMA-Factory + FSDP2.

Quickstart — serve + submit a PDF or LaTeX source dir

Easiest path uses the PaperScorer orchestrator (paperprep + scoring server + web UI all wired up). Clone, run setup, and chain into the UI:

git clone https://github.com/zlab-princeton/PaperScorer.git
cd PaperScorer
uv tool install .              # installs the `paperscorer` CLI globally on PATH
paperscorer setup --serve        # in the wizard, pick: size=3B · modality=vision · domain=arxiv
# → web UI on http://localhost:8003 (PDF upload + LaTeX dir browse)

Or hit the API directly (FastAPI on the same port):

# Submit an anonymized PDF; poll for the verdict
JOB=$(curl -s -F [email protected] http://localhost:8003/submit | jq -r .job_id)
curl http://localhost:8003/status/$JOB        # → job dict: state, verdict, p_accept, ...

# Submit a LaTeX source directory (anonymized) or an arXiv id
curl -X POST http://localhost:8003/submit_latex \
     -H "Content-Type: application/json" \
     -d '{"path": "/abs/path/to/anonymized_latex_dir"}'
curl -X POST http://localhost:8003/submit_arxiv \
     -H "Content-Type: application/json" \
     -d '{"arxiv_id": "2511.08364"}'

Headless one-shot (no server):

paperscorer run /abs/path/to/anonymized.pdf

Lower-level: stand up just a vLLM scoring server with pre-prep'd sharegpt rows (skips paperprep):

vllm serve skonan/PaperScorer-3B-Vision-arXiv --task generate --gpu-memory-utilization 0.85
# OpenAI-compat API on :8000 — format prompts per the "Prompt format" section below.

Test results (in-distribution)

Evaluated on arxiv-balanced-test (2024+ papers). The prediction is the model's own decision: Accept when the decision-token log-odds ℓ = logprob(Accept) − logprob(Reject) is ≥ 0, which is the same as parsing the \boxed{} answer. No threshold is tuned. pA = predicted accept rate; A_rec / R_rec = accept / reject recall.

Calibration (temperature scaling with a base-rate term)

ℓ measures the evidence for Accept under the balanced 50/50 prior the model was trained on. To turn it into a calibrated chance of acceptance at a venue's acceptance rate π, divide by the temperature τ = 1.702 and add the prior log-odds:

p_accept = σ(ℓ / τ + log(π / (1 − π))), with τ = 1.702 and π = 0.232 for this model.

import math
ell = logprob_accept - logprob_reject   # at the decision token; use -50 for a token missing from the top-5

def p_accept(ell, tau=1.702, pi=0.232):
    # pi = the venue's acceptance rate; default is this model's validation acceptance rate
    return 1 / (1 + math.exp(-(ell / tau + math.log(pi / (1 - pi)))))

p_accept(ell)            # chance of acceptance at the arXiv validation rate (π = 0.232)
p_accept(ell, pi=0.5)    # balanced 50/50 pool: the shift vanishes, giving σ(ℓ / τ)

τ was fit by negative log-likelihood on the natural-acceptance-rate validation split (23.2% accepts), which matches deployment, where most papers are rejected. The base-rate term is inside the fit, with π set to that split's acceptance rate, so the shift handles the base rate and τ only sets how sharp the probabilities are. p_accept is the model's chance of acceptance for the paper at a venue that accepts a fraction π of submissions.

• Decisions are unchanged. The model accepts iff ℓ ≥ 0, which is exactly p_accept ≥ π. Neither τ nor π ever changes a decision, so the test results above hold as reported.
• Other venues. π = 0.232 is a mix of the source venues' acceptance rates. Plug in a single venue's acceptance rate as pi to get the chance of acceptance there, e.g. p_accept(ell, pi=0.25) for a venue that accepts 25% of submissions.
• Balanced pools. On a 50/50 pool, set π = 0.5: the shift vanishes and p_accept = σ(ℓ / τ).

Expected calibration error (10 bins): on arxiv-balanced-test (π = 0.5), 0.096 raw → 0.030 calibrated; on the natural-rate arXiv test split, 0.134 without the base-rate term → 0.034 with it. The method follows the paper's calibration section (Section 5, "Confidence tracks accuracy"; Appendix I.3, "Calibration Details"). This 3B fit uses the same procedure on this 3B model's natural-rate validation split; the paper's Table I.1 reports 7B values only.

Note on training-size asymmetry

Arxiv-trained models saw ~71k examples (4 epochs ≈ 8884 / 17724 steps). Arxiv-trained vs ICLR-trained models saw a ~3× data gap — direct comparisons should account for it.

Prompt format

Inputs are sharegpt-style 3-turn conversations: system, human, gpt. SYSTEM is the same string across all 8 PaperScorer models. USER preamble differs per training domain. Vision variants append one <image> token per page-screenshot at the end of the user message.

SYSTEM (all PaperScorer models)

You are an expert academic reviewer tasked with evaluating research papers.

ARXIV-trained USER preamble (verbatim)

I am giving you a paper submitted to a top machine-learning venue. Predict its acceptance outcome.
 - Your answer will either be: \boxed{Accept} or \boxed{Reject}
 - Note: typical top-tier ML venues have ~25-30% acceptance rates

# <PAPER TITLE>
...paper body in markdown...

Vision variant — append <image> tokens

After the markdown body, append a space-separated run of <image> placeholders, one per page screenshot (typically 7–9 for arxiv, 8–10 for iclr). The images field on the inference request is a parallel list of PNG paths.

...end of markdown body...

<image> <image> <image> <image> <image> <image> <image>

ASSISTANT (gold)

Outcome: \boxed{Accept}

or

Outcome: \boxed{Reject}

At inference, the decision logprobs at the boxed-token position (5th generated token under the qwen template) give ℓ for temperature scaling (see Calibration above); either parse the text for the decision or read logprobs directly.

Concrete example (TEXT, ARXIV-trained)

[SYSTEM]
You are an expert academic reviewer tasked with evaluating research papers.

[USER]
I am giving you a paper submitted to a top machine-learning venue. Predict its acceptance outcome.
 - Your answer will either be: \boxed{Accept} or \boxed{Reject}
 - Note: typical top-tier ML venues have ~25-30% acceptance rates

# SSAST: SELF-SUPERVISED AUDIO SPECTROGRAM TRANSFORMER

## Abstract
... ~32k chars of paper body ...

[ASSISTANT]
Outcome: \boxed{Accept}

Concrete example (VISION, ICLR-trained)

[SYSTEM]
You are an expert academic reviewer tasked with evaluating research papers.

[USER]
I am giving you a paper. I want to predict its acceptance outcome at ICLR.
 - Your answer will either be: \boxed{Accept} or \boxed{Reject}
 - Note: ICLR generally has a ~30% acceptance rate

# ROBUST TRAINING WITH ENSEMBLE CONSENSUS

## Abstract
... ~1k chars of paper body (vision body is much shorter than text body) ...

<image> <image> <image> <image> <image> <image> <image> <image> <image>

[ASSISTANT]
Outcome: \boxed{Reject}

(With images = [page_1.png, page_2.png, …, page_9.png] on the request.)

Related models + datasets in the PaperScorer collection

All 8 single-domain SFT models (this one plus 7 siblings) plus the companion PaperScorer-Text and PaperScorer-Vision datasets live in the PaperScorer collection. Pairwise comparisons across {3B, 7B} × {text, vision} × {arxiv, openreview-iclr} are intended.

Citation

If you use PaperScorer, please cite our NeurIPS 2026 paper:

@inproceedings{konan2026paperscorer,
  title     = {PaperScorer: How Predictable Is Paper Acceptance?},
  author    = {Konan, Sachin and Liu, Jonathan and Liu, Zhuang},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}