Florence-Chan-2.1

Florence-Chan-2.1

A Florence-2-large-derived anime caption model, baked into a standalone BF16 safetensors checkpoint. No adapter is required at inference time.

What changed

• Based on the v2 cumulative teacher line and its follow-up SFT continuation.
• Extended with the latest tranche plus an equal-sized v2 retention mix.
• Kept the standard Florence 2 large baseline behavior close while improving the validated follow-up row set.
• The training lineage used GLM-4.6V and Grok-4.3 teacher captions.

Training data

• Broader cumulative teacher lineage: 44,647 rows.
• Follow-up SFT used for this checkpoint: 980 rows.
• Follow-up SFT composition: 490 latest-tranche rows + 490 retention rows.

Package

• model: Florence-Chan-2.1
• parent model: Florence 2 large
• follow-up SFT size: 980 image-caption rows
• cumulative teacher lineage size: 44,647 rows
• base model: microsoft/Florence-2-large
• adapter source: /data/anime-captioner/checkpoints/florence2_cumulative_teacher_v2_extension_late_stage_v1/best
• merged checkpoint: /data/anime-captioner/hf.co/florence2-cumulative-teacher-v2p1
• safetensors: /data/anime-captioner/hf.co/florence2-cumulative-teacher-v2p1/model.safetensors

Safe comparison set

The table below compares the baked model against the standard Florence 2 large baseline and the raw WD14 tags on eight safe, diverse images.

dataset/danbooru_images/danbooru_8428374.jpg

dataset/danbooru_images/danbooru_5341516.jpg

dataset/danbooru_images/danbooru_5294973.jpg

dataset/danbooru_images/danbooru_7019581.jpg

dataset/danbooru_images/danbooru_8654121.jpg

dataset/danbooru_images/danbooru_5221484.jpg

dataset/danbooru_images/danbooru_8643115.jpg

dataset/danbooru_images/danbooru_6621403.jpg

Notes

• Images are included at 512px for table display.
• Baseline is the standard Florence-2-large captioning path.
• WD14 tags are shown as the raw local tagger output for the same images.
• The checkpoint is merged and serialized as safetensors, so there is no adapter dependency.

Run with Transformers

Known-good runtime: transformers==4.46.3. The standard Florence-2 processor/model API works directly against the baked checkpoint.

from PIL import Image
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "/data/anime-captioner/hf.co/florence2-cumulative-teacher-v2p1"
image_path = "/data/anime-captioner/dataset/danbooru_images/danbooru_8428374.jpg"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

image = Image.open(image_path).convert("RGB")

for task in ["<CAPTION>", "<DETAILED_CAPTION>"]:
    inputs = processor(text=task, images=image, return_tensors="pt")
    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.no_grad():
        generated_ids = model.generate(
            input_ids=inputs["input_ids"],
            pixel_values=inputs["pixel_values"],
            max_new_tokens=128,
            do_sample=False,
            num_beams=1,
        )

    text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
    print(task, text)