sci-image-markdown-qwen2.5-vl-3b

Sci-Image-Markdown: Qwen2.5-VL-3B LoRA Table Extractor

Fine-tuned PEFT LoRA adapter for Qwen2.5-VL-3B-Instruct designed to extract quantitative plotted data from scientific figure panels directly into clean, structured Markdown tables.

This model was trained using an ICDAR Metric-Aware Loss (combining Differentiable Token-Weighted Numerical & Structural loss with fast in-VRAM GPU rewards for Table Edit Distance and Cell RMS Error).

📊 Benchmark Results (Test Set - 373 Scientific Figures)

---

🚀 Quickstart & Usage

Installation

pip install torch transformers peft qwen-vl-utils pillow torchvision bitsandbytes

Inference Example

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfig
from peft import PeftModel
from qwen_vl_utils import process_vision_info

model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_id = "lucasoc/sci-image-markdown-qwen2.5-vl-3b"

# 1. Load processor
processor = AutoProcessor.from_pretrained(adapter_id)

# 2. Load 4-bit base model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)
base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 3. Load trained LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 4. Extract table from figure
image_path = "figure.png"
image = Image.open(image_path).convert("RGB")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "Extract the plotted quantitative data into a clean Markdown table with column headers."}
        ]
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to("cuda")

with torch.inference_mode():
    generated_ids = model.generate(**inputs, max_new_tokens=1024, temperature=0.0)
    generated_ids_trimmed = [
        out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]
    output_text = processor.batch_decode(
        generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )[0]

print("--- Extracted Markdown Table ---")
print(output_text)

---

🛠 Training Details

• Base Architecture: Qwen2.5-VL-3B-Instruct
• Method: 4-bit QLoRA ($r=16, \alpha=32$)
• Hardware: Single NVIDIA GeForce GTX 1660 (6 GB VRAM)
• Loss: Token-Weighted ICDAR Loss (Proxy for Tree Edit Distance and Relative Numerical Error)
• Precision: 4-bit NF4 base weights, FP16 LoRA compute

📜 Citation & License

• Base Model License: Qwen Research License / Apache 2.0
• Codebase: sci-image-markdown