qwen3vl-caption-imregis

Qwen3-VL-4B — ImRegis Image Captioning

This is a Qwen3-VL-4B-Instruct model fine-tuned for Portuguese image captioning of figures extracted from geological documents. It was trained with LoRA on the ImRegis dataset, and the adapter has been merged into the base weights, so the model is standalone and can be loaded directly without PEFT.

Model description

• Base model: Qwen/Qwen3-VL-4B-Instruct
• Task: image-to-text (captioning) in Portuguese
• Training data: ImRegis — figure/box crops from geological reports paired with Portuguese captions
• Method: LoRA fine-tuning, then merged into the base model
• Languages: Portuguese (pt)

The model takes a single image and produces a descriptive caption in Portuguese. It is intended for describing figures, maps, cross-sections, and diagrams found in geoscience documents.

Intended uses & limitations

Generating Portuguese captions for figures extracted from geological documents, in line with the kind of content present in ImRegis.

Limitations.

• The model was trained on a domain-specific corpus (geological document figures); performance on out-of-domain images may be lower.
• Captions may contain factual errors or hallucinations and should be reviewed before being used in any downstream product.
• Training images were resized so the longest side is at most 448 px.

How to use

import torch
from PIL import Image
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
REPO_ID = "Geologi/qwen3vl-caption-imregis"
model = Qwen3VLForConditionalGeneration.from_pretrained(
    REPO_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(REPO_ID)
image = Image.open("example.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Descreva esta imagem em português"},
    ],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=128)
print(processor.batch_decode(output, skip_special_tokens=True)[0])

Because the LoRA adapter is already merged, no peft import is required.