food-recognition

DINOv3-food

DINOv3-food is a food image recognition model fine-tuned from facebook/dinov3-vitl16-pretrain-lvd1689m on TSOTSA-Img, a merged food image dataset built from AFD, FruitVeg-81, Food-101, and UECFood256.

The model predicts 389 food categories and uses a DINOv3 ViT-L/16 backbone with a lightweight linear classification head.

Dataset

TSOTSA-Img is the merged dataset used for food recognition in this work. It is split into training and test subsets:

• Training split: used to fine-tune the model.
• Test split: used for final evaluation.

The merged dataset combines food images and labels from:

• AFD
• FruitVeg-81
• Food-101
• UECFood256

Training

The selected checkpoint was fine-tuned for 8 epochs.

Validation metrics for the selected run:

Evaluation

Final evaluation was performed on the individual source datasets and on the merged TSOTSA-Img test split.

For the TSOTSA-Img test split:

Model format

This repository stores a custom backbone-plus-classifier model:

• backbone/: DINOv3 backbone saved with transformers.
• classifier.pt: linear classification head.
• classifier_config.json: label mappings and classifier metadata.
• preprocessor_config.json: image preprocessing configuration.

Because this model uses a custom wrapper around the DINOv3 backbone, loading it with AutoModelForImageClassification.from_pretrained(...) is not sufficient. Use the project loader or reconstruct the wrapper before inference.

Usage

Example with the project inference class:

from inference.food_classifier import FoodClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
classifier = FoodClassifier(model_dir)

prediction = classifier.predict("path/to/food_image.jpg")
print(prediction)

Manual loading:

import json
import torch
from transformers import AutoImageProcessor, AutoModel
from finetuning.train_classifier import BackboneImageClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"

with open(f"{model_dir}/classifier_config.json", "r", encoding="utf-8") as f:
    classifier_config = json.load(f)

id2label = {
    int(label_id): label
    for label_id, label in classifier_config["id2label"].items()
}
label2id = {
    label: int(label_id)
    for label, label_id in classifier_config["label2id"].items()
}

backbone = AutoModel.from_pretrained(f"{model_dir}/backbone")
model = BackboneImageClassifier(
    backbone=backbone,
    num_labels=int(classifier_config["num_labels"]),
    id2label=id2label,
    label2id=label2id,
)

classifier_state = torch.load(f"{model_dir}/classifier.pt", map_location="cpu")
model.classifier.load_state_dict(classifier_state)
model.eval()

processor = AutoImageProcessor.from_pretrained(model_dir)

Intended use

This model is intended for food image recognition over the TSOTSA-Img label space. It can be used for research experiments, dataset benchmarking, and food recognition pipelines where the target labels overlap with the 389 supported categories.

Limitations

• The model is restricted to the 389 labels in classifier_config.json.
• Performance may degrade on food categories outside the TSOTSA-Img label space.
• Predictions may be sensitive to ambiguous images, mixed dishes, heavy occlusion, or visually similar food categories.
• The model card reports accuracy on the available benchmark splits and should not be interpreted as performance on all possible food domains.

Citation