florence-large-lora-v22

Florence-2-large LoRA v22 — UI grounding

PEFT LoRA adapter for Florence-2-large-ft, iteration v22 of the training pipeline. Published for reproducibility; not recommended for production use (see Limitations).

Benchmark results — partial coverage

• 91% pass rate (10/11) on a covered subset of the Magnitude suite — but coverage is intentionally narrow: only easy tests (#1, #4, #5, #8–14) plus #15 (Wikipedia history radio, the structural fail-for-everyone test). Not directly comparable to other variants which were run on the full 50.
• On the same 11-test subset Khabner/florence-large-lora-v1 gets 9/11 (82%), so v22 looks better in this narrow window — but the absolute number isn't a like-for-like benchmark of the full distribution.
• Median best-pass time 26.8s, mean 43.6s — fastest in the family, again partially due to the easy subset.

Limitations

Same as the other Florence variants: structural failures on [show] toggles, tiny [3] superscripts, and dense-table-row controls. Larger backbone doesn't lift these.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoProcessor

base = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "Khabner/florence-large-lora-v22").eval()

Reference: github.com/VLM-WEBTEST/magnitude_integration.