judgebench-qwen-lora-judge

judgecliff Qwen2.5-VL-7B LoRA brand judge (J3b)

Part of judgecliff: which image-QC judges survive optimization pressure?

Base: Qwen/Qwen2.5-VL-7B-Instruct. This is a LoRA adapter on the LM attention layers (vision tower frozen), trained with binary brand supervision on the same train split as the SigLIP-tuned judges: rhode images = "yes", competitor-brand images = "no". No violation negatives.

Scoring: the judge score is P("yes") — the probability of the "yes" token as the first generated token, given the image and this prompt:

You are a brand-consistency evaluator. Is this image consistent with the visual identity of rhode, the beauty brand? Answer with exactly one word: yes or no.

Report card (vs the zero-shot base): fine-tuning cured the name-tag shortcut — logo-masking delta 0.31 → 0.05, masked-set brand AUC 0.67 → 0.93 — but collapsed ordinal ranking (brand-dial Spearman 0.94 → 0.16; 34% of items score >0.99). Violation detection weak (0.12 mean) but uniform. Calibration ECE 0.20 → 0.03; temporal holdout AUC 0.99.

Load with peft.PeftModel.from_pretrained over the base model. Full protocol and findings: https://github.com/amargupta0428/judgecliff.