FoodExtract-Vision-SmolVLM2-500M-fine-tune

FoodExtract-Vision-SmolVLM2-500M

A fine-tuned Vision-Language Model for structured food and drink extraction from images. Given an input image, the model outputs a structured JSON containing food classification, image title, and extracted food/drink items.

Model Description

Intended Use

This model is designed for:

• 🍕 Food/Drink Classification: Determine if an image contains food or drinks
• 📝 Structured Data Extraction: Extract food and drink items into JSON format
• 🏷️ Image Captioning: Generate food-related titles for images

Output Format

{
  "is_food": 1,
  "image_title": "macaron assortment",
  "food_items": ["yellow macaron", "white macaron", "green macaron"],
  "drink_items": []
}

Quick start

from transformers import pipeline
import torch

# Load the fine-tuned model
pipe = pipeline(
    "image-text-to-text",
    model="CreatorJarvis/FoodExtract-Vision-SmolVLM2-500M-fine-tune",
    dtype=torch.bfloat16,
    device_map="auto"
)

# Prepare input message
message = [{
    "role": "user",
    "content": [
        {"type": "image", "image": your_image},  # PIL.Image object
        {"type": "text", "text": """Classify the given input image into food or not and if edible food or drink items are present, extract those to a list. If no food/drink items are visible, return empty lists.

Only return valid JSON in the following form:

```json
{
  'is_food': 0,
  'image_title': '',
  'food_items': [],
  'drink_items': []
}
```} ] }]

Training Details

Dataset

Dataset Source: mrdbourke/FoodExtract-1k-Vision

Training Configuration

Training Strategy

The Vision Encoder was frozen during training to:

• Preserve pre-trained visual representations
• Reduce trainable parameters and memory usage
• Improve training stability on small datasets
• Mitigate overfitting

This approach is inspired by the SmolDocling paper.

Training Results

Final Training Loss: 0.0518

Experiment Tracking

<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="30"/>

Demo

Try the model on Hugging Face Spaces:

🚀 FoodExtract-Vision Demo

The demo compares outputs from the base model vs. the fine-tuned model side-by-side.

Limitations

• Trained on a relatively small dataset (1.5k images)
• May struggle with complex multi-item food scenes
• Occasional repetitive generation patterns
• Best performance on single-dish food images

Framework Versions

Citation

If you use this model, please cite:

@misc{foodextract-vision-2025,
  title        = {FoodExtract-Vision: Fine-tuned SmolVLM2 for Structured Food Extraction},
  author       = {Jarvis Zhang},
  year         = 2025,
  publisher    = {Hugging Face},
  howpublished = {\url{[https://huggingface.co/CreatorJarvis/FoodExtract-Vision-SmolVLM2-500M-fine-tune](https://huggingface.co/CreatorJarvis/FoodExtract-Vision-SmolVLM2-500M-fine-tune)}}
}