Downloads · 30 days
6
14% of all-time downloads
rukiyeberna/qwen2.5-vl-3b-food-extract-lora
qwen2.5-vl-3b-food-extract-lora is a image-text-to-text model from rukiyeberna. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
This repository contains the parameter-efficient LoRA adapter weights for Qwen2.5-VL-3B-Instruct, fine-tuned for multimodal structured attribute extraction from food photography. The model analyzes dish images and out…
Downloads · 30 days
6
14% of all-time downloads
All-time downloads
43
Public
Repo size
205 MB
Likes
1
Public
Click a slice to open those files.
.safetensors120 MB · 59%
From the Hugging Face model README
This repository contains the parameter-efficient LoRA adapter weights for Qwen2.5-VL-3B-Instruct, fine-tuned for multimodal structured attribute extraction from food photography. The model analyzes dish images and outputs strictly formatted JSON including food presence verification, title, ingredients, and accompanying beverages.
Qwen/Qwen2.5-VL-3B-InstructZero-shot and structured visual extraction of culinary attributes from food images. Designed to produce strict JSON schemas for menu parsing, dietary tracking, recipe indexing, and food dataset labeling.
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER_PATH = "rukiyeberna/qwen2.5-vl-3b-food-extract-lora"
# Load base model & LoRA adapter
base_model = AutoModelForImageTextToText.from_pretrained(
BASE_MODEL,
torch_dtype=torch.bfloat16,
device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
processor = AutoProcessor.from_pretrained(ADAPTER_PATH)
model.eval()
# Load image and build prompt
image = Image.open("sample_food.jpg").convert("RGB")
prompt = """Analyze this image and return only valid JSON with exactly these keys:
- is_food
- image_title
- food_items
- drink_items"""
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": prompt},
],
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)
with torch.inference_mode():
generated_ids = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False,
repetition_penalty=1.15,
no_repeat_ngram_size=3
)
output = processor.batch_decode(
generated_ids[:, inputs.input_ids.shape[1]:],
skip_special_tokens=True
)[0]
print(output)
Trained on mrdbourke/FoodExtract-1k-Vision (1,510 training samples) preprocessed into LLaVA-style conversational format (image + structured JSON response).
bfloat16[cite: 1]q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj[cite: 1]1.7121 $\rightarrow$ Final 0.8888 (48.08% convergence reduction)[cite: 1]Evaluated on unseen food test images across structured schema integrity, sequence similarity, and fuzzy set-based overlap[cite: 1]:
is_food Classification Accuracy[cite: 1]