florence2-base-0.5b-detection

Florence-2-base-0.5B — beats Qwen3-VL-4B on COCO detection

A 0.23B vision-language model (Florence-2-base-ft, fine-tuned) that beats the ~9×-larger Qwen3-VL-4B-Instruct on exhaustive COCO object detection — well under a 0.5B budget.

Headline result (identical images, identical harness, pycocotools)

+0.048 mAP (+12% relative) vs a model ~9× its size, on the same 150 val images with a byte-identical parser/matcher/scorer. Driven by Florence-2's detection-native architecture (high recall, precise boxes) vs a generalist VLM's low-recall autoregressive box output.

Full honest head-to-head vs Qwen3-VL-4B

Scope (honest): this model wins object detection. It does not beat the 4B on VQA or referring-grounding — a 0.5B model has capacity for one strong capability, not both. Detection is where a small detection-native model genuinely outperforms a much larger generalist. (Established across ~10 experiments and both architectures; chat VLMs of this size get COCO mAP ~0.03.)

Usage

import torch
from transformers import Florence2ForConditionalGeneration, AutoProcessor
from PIL import Image

model_id = "neuromorphic-research/florence2-base-0.5b-detection"
model = Florence2ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
proc = AutoProcessor.from_pretrained(model_id)

image = Image.open("image.jpg").convert("RGB")
inputs = proc(text="<OD>", images=image, return_tensors="pt").to("cuda", torch.bfloat16)
out = model.generate(input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"],
                     max_new_tokens=1024, num_beams=3)
text = proc.batch_decode(out, skip_special_tokens=False)[0]
print(proc.post_process_generation(text, task="<OD>", image_size=image.size))

Also supports <OPEN_VOCABULARY_DETECTION>, <CAPTION_TO_PHRASE_GROUNDING>, and short-answer VQA.

Training

Fine-tuned from florence-community/Florence-2-base-ft on a multi-task mix: COCO detection (all-class + per-class), RefCOCO/+/g grounding, VQAv2, GQA, TallyQA, PixMo-Count. bf16, LR 4e-6 (vision tower 2e-6), gradient checkpointing. License: MIT (inherits Florence-2-base-ft).