Downloads · 30 days
24
27% of all-time downloads
MayaKD/qwen2-vl-7b-nutrition-vllm
qwen2-vl-7b-nutrition-vllm is a image-text-to-text model from MayaKD. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Fine-tuned Qwen/Qwen2-VL-7B-Instruct that localizes nutrition tables in product photographs. Given an image, it emits bounding boxes as text using Qwen2-VL's native grounding tokens — no detection head, no anchor boxes.
Downloads · 30 days
24
27% of all-time downloads
All-time downloads
88
Public
Parameters
8.3B
33.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.6 GB · 100%
From the Hugging Face model README
Fine-tuned Qwen/Qwen2-VL-7B-Instruct
that localizes nutrition tables in product photographs. Given an image, it emits bounding
boxes as text using Qwen2-VL's native grounding tokens — no detection head, no anchor boxes.
These are merged full weights at the repo root, so vLLM and from_pretrained can load
this repo directly. For training artifacts (LoRA adapters, per-stage checkpoints, optimizer
state) see MayaKD/qwen2-vl-7b-nutrition.
Evaluated on all 123 validation samples of openfoodfacts/nutrition-table-detection:
| Metric | Value | |
|---|---|---|
| Mean IoU | 0.82 | primary metric |
| [email protected] | ≤ 0.91 | upper bound |
| [email protected] | ≤ 0.89 | upper bound |
| [email protected] | ≤ 0.90 | upper bound |
Why the threshold metrics are bounds. They were computed with a matching rule that counted every IoU-matrix cell above 0.5 rather than pairing each ground-truth box with at most one prediction, so duplicate detections on one table overcounted true positives. The corrected implementation does greedy one-to-one matching; the bias it removes is strictly upward, so true values sit at or below those shown. Mean IoU is threshold-free, never used that path, and is the metric to judge this model on.
Serving throughput on a single GPU, measured with real async concurrency:
| Concurrency | Mean latency | P95 | Throughput |
|---|---|---|---|
| 1 | 1,378 ms | 1,506 ms | 0.73 req/s |
| 4 | 399 ms | 467 ms | 9.89 req/s |
| 8 | 435 ms | 1,056 ms | 17.82 req/s |
| 16 | 16,528 ms | 22,653 ms | 0.94 req/s |
Throughput peaks at c=8 and collapses at c=16 as the scheduler saturates.
Send the strings the model was fine-tuned on, exactly as written below. How much accuracy is lost by deviating has not been measured cleanly, so treat these as part of the interface rather than as tunable inputs.
system: You are a Vision Language Model specialized in interpreting visual data from product images.
Your task is to analyze the provided product images and detect the nutrition tables in a certain format.
Focus on delivering accurate, succinct answers based on the visual information. Avoid additional explanation unless absolutely necessary.
user: Detect the bounding boxes of all nutrition tables in the image.
import torch
from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
from qwen_vl_utils import process_vision_info
REPO = "MayaKD/qwen2-vl-7b-nutrition-vllm"
SYSTEM = (
"You are a Vision Language Model specialized in interpreting visual data from product images.\n"
"Your task is to analyze the provided product images and detect the nutrition tables in a certain format.\n"
"Focus on delivering accurate, succinct answers based on the visual information. "
"Avoid additional explanation unless absolutely necessary."
)
PROMPT = "Detect the bounding boxes of all nutrition tables in the image."
model = Qwen2VLForConditionalGeneration.from_pretrained(
REPO, dtype=torch.bfloat16, device_map="auto",
)
processor = Qwen2VLProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "image", "image": "https://static.openfoodfacts.org/images/products/27563564/2.jpg"},
{"type": "text", "text": PROMPT},
]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
out_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
trimmed = [o[len(i):] for i, o in zip(inputs.input_ids, out_ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=False)[0])
python -m vllm.entrypoints.openai.api_server \
--model MayaKD/qwen2-vl-7b-nutrition-vllm \
--dtype bfloat16 --host 0.0.0.0 --port 8000
Each detection is emitted as:
<|object_ref_start|>nutrition-table<|object_ref_end|><|box_start|>(x0, y0),(x1, y1)<|box_end|>
Coordinates are (x_min, y_min), (x_max, y_max) on a 0–1000 scale, relative to image
dimensions. Multiple detections concatenate. Images with no table produce "No table found."
LoRA (r=16, α=32) applied to the frozen base in two stages. Adapters are merged into the base between stages, so each stage starts from clean full weights rather than stacked adapters.
| Stage | Targets | Epochs | LR |
|---|---|---|---|
| 1 — Full vision | All vision blocks (attn + MLP) + merger MLP | 6 | 1e-4 |
| 2 — Joint | Vision + LLM q/k/v/o_proj, gate/up/down_proj | 6 | 1e-5 |
Trained with SFTTrainer + accelerate on 2× RTX Pro 6000, BF16 with Flash Attention 2 and
fused AdamW. Best checkpoint by eval_loss, early stopping patience 3. Image resolution
min_pixels=784, max_pixels=705600; sequence length is deliberately not capped, since
truncation corrupts image tokens.
A three-stage variant reached 0.893 mean IoU in exploratory runs, indicating headroom above this checkpoint. Those weights were not preserved; this two-stage model is the reproducible artifact.
MayaKD/qwen2-vl-7b-nutritionopenfoodfacts/nutrition-table-detectionApache-2.0, inherited from Qwen2-VL-7B-Instruct. Project code is MIT.