CalorieK

CalorieK — food-photo calorie & macro estimation

A single RGB food photo in, five numbers out: calories (kcal), mass (g), protein (g), carbohydrate (g), fat (g). CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluated under a strict, leakage-audited Nutrition5k protocol.

Headline result — locked Nutrition5k test set (606 dishes, official test split, never seen)

CalorieK more than halves CalorieCLIP's calorie MAE and nearly doubles its within-±100 hit rate on the same dishes. Every test image is a deterministic central frame from the official Nutrition5k camera-A side-angle video of an official test-split dish — selected by dish ID, never by image quality. Evaluation is dish-level (physical_meal_id).

Two of the 608 official test dishes carry corrupt ground-truth labels (a single 7,974 g "olives" ingredient gives one dish 876 g of fat in a 159 g dish, and another an 8 kg mass) and are excluded as physically impossible (calories > 2000). Including them, CalorieK's calorie MAE is 86.2 kcal (RMSE 533, driven almost entirely by those two dishes); excluding them RMSE is 94.

Macronutrient accuracy (same 606 dishes)

Comparison with open VLMs (paired 303-dish subset, seed 17)

Each model answered the same calorie question on the same dishes, scored against the official Nutrition5k calorie labels.

‡ Qwen3.5-9B: 8-bit GGUF (Q8_0) via llama.cpp, 4k context, thinking off. § MiniMax M3: hosted reasoning model, chain-of-thought enabled — extensive per-dish reasoning did not help; it lands last with a strong positive bias. Reasoning does not fix calibration.

Out-of-distribution comparison on a fresh MM-Food-100K sample

To stress-test both models outside the Nutrition5k training distribution, we built a second, disjoint OOD benchmark from the public MM-Food-100K dataset (Humanbased-AI/MM-Food-100K). 500 fresh images were downloaded from the dataset's original b18a.io URLs — none of them appear in the prior 500-dish OOD subset (which used a different scratchpad cache), and none appear in CalorieK's training data. The food-type mix was deliberately flipped from the prior test (238/177/42/42/1 Homemade/Restaurant/Raw/Packaged/Other → 100/0/200/199/0 Homemade/Raw/Packaged/Other), so this benchmark targets the food categories the prior test barely touched.

All per-image labels come straight from the published MM-Food-100K CSV (calories_kcal, protein_g, carbohydrate_g, fat_g per row) — no per-class nutrition lookup, no fabrication. Mean target 233 kcal, range 2–2500 kcal. Seed 137 for the sampling. Full reproducibility: scripts/sample_ood_v2.py → scripts/download_ood_v2.py → scripts/build_ood_v2_manifest.py → scripts/compare_ood.py.

Per food-type (where the new mix actually shifts the failure modes):

CalorieK wins on Homemade and Packaged by 80 and 81 kcal MAE respectively, and almost doubles CalorieCLIP's within-±100 hit rate on those categories. On Raw vegetables & fruit both models fall apart (MAE 234–244 on a mean target of ~150 kcal, within-±100 under 20%) — this is the long-tail small-portion regime that neither model handles, and the scatter shows both regress to the 200–400 kcal range even for items like a single 62 kcal "Orange Slices" or 68 kcal "Fresh Peaches." Four outliers in the 1,800–2,500 kcal range (e.g. "Pork Belly" labelled 2500 kcal and categorised as Raw vegetables) drag both models' overall MAE up; these are the rare cases where the MM-Food-100K ground truth is implausible (a single 2,000 g "Pork Belly" portion is plausible, but its food_type labelling is clearly wrong). CalorieCLIP is also systematically over-confident: it predicts 760 kcal on a 68 kcal peach and 771 kcal on a 150 kcal dairy drink, a +156 kcal bias against CalorieK's +82.

Caveats on this benchmark. (1) MM-Food-100K is the same source dataset as the prior OOD test, just a fresh, disjoint sample — it is not a wholly new dataset. We could not find a third, truly-distinct, downloadable per-image-calorie food dataset (OmniFood8K is author-only via Google Drive; CGMacros is CGM-centric; the only other plausible option, Food-101, has class labels not per-image calories and would have required a fabricated per-class nutrition table, which violates our no-fabrication rule). (2) MM-Food-100K labels are AI/crowd estimates, not weighed ground truth — a ±10–20% label noise floor is expected. (3) The 2,500 kcal outliers are clearly label noise and inflate both models' MAE.

Full per-prediction CSV: outputs/compare_ood_v2/predictions.csv. Aggregate metrics: outputs/compare_ood_v2/report.json. Raw downloads: data/mmfood100k_v2/images/.

Comparison with open-source calorie VLMs (same protocol)

Beyond CalorieCLIP, we ran the published checkpoints of every other public calorie VLM we could get to run end-to-end on a single RGB photo — Food-R1 (8-bit), Boba 0.8B (Q4_K_M GGUF), DPF-Nutrition, and OmniFood FAFM — under the same locked protocol and the same MM-Food-100K v2 OOD sample, so the numbers are directly comparable. The full per-model report (including the per-task macro/mass errors) lives in outputs/external_model_comparison.md.

Locked Nutrition5k test (606 dishes; 2 corrupt-label dishes excluded)

Food-R1 and Boba each had one image (dish_1565810969) where the model entered a non-terminating token loop; their coverage is 99.83% on 608. All other models cover 100%.

MM-Food-100K v2 transfer — 499 disjoint images (OOD)

DPF-Nutrition wins OOD by 3.5 kcal — inside the noise band, not a decisive lead without a paired bootstrap. The locked-test gap (56.3 vs 77.0 for the next-best public model) is the cleaner signal: under an identical, in-distribution protocol, CalorieK's calorie MAE is 27% lower than the next open-source VLM.

What "same protocol" means. Same camera-A center frame per dish as the locked headline test; same scalar calorie target from the official Nutrition5k dish metadata; same MM-Food-100K v2 manifest for OOD; same JSON-completeness and memory controls for the GGUF runner (scripts/benchmark_gguf_food.py); and the published PMAE / MAE definitions, not the paper authors' own splits. Public PyTorch adapters: src/ccalorie/external_bench.py. Locked predictions: outputs/external_bench_locked/, outputs/boba_locked/. OOD predictions: outputs/external_bench_mmfood/, outputs/boba_mmfood/.

Training method

• Architecture: google/siglip2-base-patch16-224 vision tower (92M params) → LayerNorm → MLP head (768→768→256→5) with softplus outputs. Trained with composite Huber loss, per-task weighted.
• Augmentation: MixUp + CutMix (both α=0.4, 50% probability per batch), plus standard RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter.
• Schedule: 20 epochs × 20,000 samples, effective batch 256 (bf16), head LR 1e-4 / trunk LR 2e-5, cosine schedule with restarts, 5% warmup, first 5% head-only, last 4 vision blocks unfrozen, weight decay 0.05, seed 17.
• Checkpoint selection: best validation calorie MAE on 80 held-out camera-A dishes (validation MAE 57.3 kcal). Training ran ~13 minutes on one RTX 4080 (16 GB).
• Calibration: a conformal radius fitted on 86 held-out calibration dishes gives 90% prediction intervals of ±119 kcal (calibration.json).

Training data (20,000 samples, leakage-audited)

Nutrition labels are taken directly from the official Nutrition5k dish metadata. Leakage controls, all enforced programmatically before training:

• Group separation: splits partition by physical dish (physical_meal_id); no dish appears in two splits.
• Exact-media: SHA-256 disjointness across fit / validation / calibration / test.
• Near-duplicate: perceptual-hash disjointness; 2 fit images visually colliding with held-out frames were dropped and backfilled.
• Official-split proof: a Nutrition5k registry maps nutrition signatures to official train/test IDs; ambiguous signatures are excluded. The audit trail (SHA-256 of every manifest and the source tree) ships in the results.

Usage

import torch, torch.nn as nn, torch.nn.functional as F
from transformers import AutoImageProcessor, SiglipVisionModel
from huggingface_hub import hf_hub_download
from PIL import Image

class NutritionHead(nn.Module):
    def __init__(self, input_dim: int = 768):
        super().__init__()
        self.net = nn.Sequential(
            nn.LayerNorm(input_dim), nn.Linear(input_dim, 768), nn.GELU(), nn.Dropout(.15),
            nn.Linear(768, 256), nn.GELU(), nn.Dropout(.10), nn.Linear(256, 5))
    def forward(self, x): return F.softplus(self.net(x))

MODEL_ID = "google/siglip2-base-patch16-224"
processor = AutoImageProcessor.from_pretrained(MODEL_ID)
vision = SiglipVisionModel.from_pretrained(MODEL_ID)
head = NutritionHead()
state = torch.load(hf_hub_download("Dralkh/CalorieK", "caloriek.pt"), map_location="cpu", weights_only=False)
vision.load_state_dict(state["vision_state"]); head.load_state_dict(state["head_state"])
vision.eval(); head.eval()

image = Image.open("meal.jpg").convert("RGB")
pixels = processor(images=image, return_tensors="pt").pixel_values
with torch.inference_mode():
    out = vision(pixel_values=pixels)
    feats = out.pooler_output if out.pooler_output is not None else out.last_hidden_state[:, 0]
    kcal, mass_g, protein_g, carb_g, fat_g = head(feats.float())[0].tolist()
print(f"{kcal:.0f} kcal ± 119 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/C{carb_g:.0f}/F{fat_g:.0f}")

Limitations

• Domain: trained and evaluated on Nutrition5k-style plated single dishes photographed from the side; accuracy on multi-plate scenes, packaged food, or drinks is untested.
• High-calorie under-prediction: the model plateaus and under-predicts on dishes above ~500 kcal (see the scatter above). Use the calibrated interval, not the point estimate, for anything that matters.
• Frames, not meals: training frames come from videos of ~1,250 distinct dishes; visual diversity is narrower than the sample count suggests.

Citation

@misc{caloriek2026,
  title={CalorieK: Accurate Food Calorie Estimation from a Single RGB Image},
  author={Faisal Alkharji},
  year={2026},
  url={https://huggingface.co/Dralkh/CalorieK}
}