Downloads · 30 days
29
9% of all-time downloads
Dralkh/CalorieK
CalorieK is a image-to-text model from Dralkh. Use it when you need a caption or text from an image. The card lists the license as mit.
A single RGB food photo in, five numbers out: calories (kcal), mass (g), protein (g), carbohydrate (g), fat (g). CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluate…
Downloads · 30 days
29
9% of all-time downloads
All-time downloads
327
Public
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.pt375 MB · 100%
From the Hugging Face model README
A single RGB food photo in, five numbers out: calories (kcal), mass (g), protein (g), carbohydrate (g), fat (g). CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluated under a strict, leakage-audited Nutrition5k protocol.

| Metric | CalorieK | CalorieCLIP (same dishes) |
|---|---|---|
| Calorie MAE | 56.3 kcal | 126.4 kcal |
| Median absolute error | 30.0 kcal | 106.2 kcal |
| Within ±50 kcal | 65.0% | 20.3% |
| Within ±100 kcal | 84.3% | 47.2% |
| Bias | −19.5 kcal | +97.5 kcal |
| Invalid predictions | 0 / 606 | 0 / 606 |
CalorieK more than halves CalorieCLIP's calorie MAE and nearly doubles its within-±100 hit rate on the
same dishes. Every test image is a deterministic central frame from the official Nutrition5k camera-A
side-angle video of an official test-split dish — selected by dish ID, never by image quality.
Evaluation is dish-level (physical_meal_id).
Two of the 608 official test dishes carry corrupt ground-truth labels (a single 7,974 g "olives" ingredient gives one dish 876 g of fat in a 159 g dish, and another an 8 kg mass) and are excluded as physically impossible (calories > 2000). Including them, CalorieK's calorie MAE is 86.2 kcal (RMSE 533, driven almost entirely by those two dishes); excluding them RMSE is 94.
| Task | MAE | PMAE |
|---|---|---|
| Mass | 55.3 g | 30.1% |
| Protein | 6.2 g | 37.4% |
| Carbohydrate | 6.3 g | 35.3% |
| Fat | 5.0 g | 43.5% |
Each model answered the same calorie question on the same dishes, scored against the official Nutrition5k calorie labels.

| Model | Params | MAE (kcal) | Median AE | Within ±100 kcal | Bias |
|---|---|---|---|---|---|
| CalorieK (ours) | 93.7M | 54.7 | 27.8 | 82.5% | −16.3 |
| Qwen3.5-9B (Q8_0)‡ | 9B | 82.6 | 55.7 | 68.6% | −14.0 |
| CalorieCLIP (public) | 151M | 126.7 | 106.4 | 46.9% | +101.0 |
| MiniMax M3 (reasoning)§ | 428B (MoE) | 130.6 | 94.7 | 52.8% | +83.7 |
‡ Qwen3.5-9B: 8-bit GGUF (Q8_0) via llama.cpp, 4k context, thinking off. § MiniMax M3: hosted reasoning model, chain-of-thought enabled — extensive per-dish reasoning did not help; it lands last with a strong positive bias. Reasoning does not fix calibration.
To stress-test both models outside the Nutrition5k training distribution, we built a second,
disjoint OOD benchmark from the public MM-Food-100K dataset
(Humanbased-AI/MM-Food-100K).
500 fresh images were downloaded from the dataset's original b18a.io URLs — none of them appear in
the prior 500-dish OOD subset (which used a different scratchpad cache), and none appear in
CalorieK's training data. The food-type mix was deliberately flipped from the prior test
(238/177/42/42/1 Homemade/Restaurant/Raw/Packaged/Other → 100/0/200/199/0 Homemade/Raw/Packaged/Other),
so this benchmark targets the food categories the prior test barely touched.
All per-image labels come straight from the published MM-Food-100K CSV (calories_kcal,
protein_g, carbohydrate_g, fat_g per row) — no per-class nutrition lookup, no fabrication.
Mean target 233 kcal, range 2–2500 kcal. Seed 137 for the sampling. Full reproducibility:
scripts/sample_ood_v2.py → scripts/download_ood_v2.py → scripts/build_ood_v2_manifest.py
→ scripts/compare_ood.py.



| Metric (n = 499) | CalorieK | CalorieCLIP | Δ (CK − CC) |
|---|---|---|---|
| Calorie MAE | 181.2 kcal | 225.3 kcal | −44.1 |
| Median AE | 155.5 | 194.6 | −39.1 |
| RMSE | 272.6 | 304.8 | −32.2 |
| Within ±50 kcal | 19.2% | 9.8% | +9.4 pp |
| Within ±100 kcal | 34.9% | 22.8% | +12.0 pp |
| Bias | +82.3 | +156.4 | −74.1 |
| Invalid predictions | 0 / 499 | 0 / 499 | — |
Per food-type (where the new mix actually shifts the failure modes):
| Food type | n | CalorieK MAE | CalorieK w/±100 | CalorieCLIP MAE | CalorieCLIP w/±100 |
|---|---|---|---|---|---|
| Homemade food | 100 | 171 | 37% | 251 | 23% |
| Packaged food | 199 | 123 | 59% | 204 | 27% |
| Raw vegetables & fruits | 200 | 244 | 10% | 234 | 19% |
CalorieK wins on Homemade and Packaged by 80 and 81 kcal MAE respectively, and almost doubles
CalorieCLIP's within-±100 hit rate on those categories. On Raw vegetables & fruit both models
fall apart (MAE 234–244 on a mean target of ~150 kcal, within-±100 under 20%) — this is the
long-tail small-portion regime that neither model handles, and the scatter shows both regress to
the 200–400 kcal range even for items like a single 62 kcal "Orange Slices" or 68 kcal "Fresh
Peaches." Four outliers in the 1,800–2,500 kcal range (e.g. "Pork Belly" labelled 2500 kcal and
categorised as Raw vegetables) drag both models' overall MAE up; these are the rare cases where
the MM-Food-100K ground truth is implausible (a single 2,000 g "Pork Belly" portion is
plausible, but its food_type labelling is clearly wrong). CalorieCLIP is also systematically
over-confident: it predicts 760 kcal on a 68 kcal peach and 771 kcal on a 150 kcal dairy drink,
a +156 kcal bias against CalorieK's +82.
Caveats on this benchmark. (1) MM-Food-100K is the same source dataset as the prior OOD test, just a fresh, disjoint sample — it is not a wholly new dataset. We could not find a third, truly-distinct, downloadable per-image-calorie food dataset (OmniFood8K is author-only via Google Drive; CGMacros is CGM-centric; the only other plausible option, Food-101, has class labels not per-image calories and would have required a fabricated per-class nutrition table, which violates our no-fabrication rule). (2) MM-Food-100K labels are AI/crowd estimates, not weighed ground truth — a ±10–20% label noise floor is expected. (3) The 2,500 kcal outliers are clearly label noise and inflate both models' MAE.
Full per-prediction CSV: outputs/compare_ood_v2/predictions.csv. Aggregate metrics:
outputs/compare_ood_v2/report.json. Raw downloads: data/mmfood100k_v2/images/.
Beyond CalorieCLIP, we ran the published checkpoints of every other public calorie VLM we could
get to run end-to-end on a single RGB photo — Food-R1 (8-bit), Boba 0.8B (Q4_K_M GGUF),
DPF-Nutrition, and OmniFood FAFM — under the same locked protocol and the same MM-Food-100K v2
OOD sample, so the numbers are directly comparable. The full per-model report (including the
per-task macro/mass errors) lives in outputs/external_model_comparison.md.

| Model | Checkpoint | n | Calorie MAE | PMAE |
|---|---|---|---|---|
| CalorieK (ours) | local | 606 | 56.3 | 24.1% |
| Food-R1 (8-bit) | zy12123/Food-R1 | 605 | 77.0 | 32.9% |
| Boba 0.8B (Q4_K_M) | opengvlab/Boba | 605 | 79.5 | 33.9% |
| DPF-Nutrition | T0MYYY/dpf-nutrition | 606 | 109.1 | 46.6% |
| CalorieCLIP | jeong-jasonji/CalorieCLIP | 606 | 126.4 | 54.0% |
| OmniFood FAFM | OmniFood8K release | 606 | 132.5 | 56.6% |
Food-R1 and Boba each had one image (dish_1565810969) where the model entered a non-terminating
token loop; their coverage is 99.83% on 608. All other models cover 100%.
| Model | OOD-eligible? | n | Calorie MAE | PMAE | Bias |
|---|---|---|---|---|---|
| DPF-Nutrition | yes | 499 | 177.7 | 76.4% | +30.8 |
| CalorieK (ours) | yes | 499 | 181.2 | — | +82.3 |
| Boba 0.8B (Q4_K_M) | yes | 499 | 193.4 | 83.1% | −24.5 |
| OmniFood FAFM | yes | 499 | 216.1 | 92.9% | +131.2 |
| CalorieCLIP | yes | 499 | 225.3 | — | +156.4 |
| Food-R1 | no | — | — | — | trained on CalorieBench-80K derived from MM-Food-100K |
DPF-Nutrition wins OOD by 3.5 kcal — inside the noise band, not a decisive lead without a paired bootstrap. The locked-test gap (56.3 vs 77.0 for the next-best public model) is the cleaner signal: under an identical, in-distribution protocol, CalorieK's calorie MAE is 27% lower than the next open-source VLM.
What "same protocol" means. Same camera-A center frame per dish as the locked headline
test; same scalar calorie target from the official Nutrition5k dish metadata; same MM-Food-100K
v2 manifest for OOD; same JSON-completeness and memory controls for the GGUF runner
(scripts/benchmark_gguf_food.py); and the published PMAE / MAE definitions, not the paper
authors' own splits. Public PyTorch adapters: src/ccalorie/external_bench.py. Locked
predictions: outputs/external_bench_locked/, outputs/boba_locked/. OOD predictions:
outputs/external_bench_mmfood/, outputs/boba_mmfood/.
google/siglip2-base-patch16-224 vision tower (92M params) → LayerNorm →
MLP head (768→768→256→5) with softplus outputs. Trained with composite Huber loss, per-task weighted.calibration.json).| Source | Samples | Notes |
|---|---|---|
pinkieseb/nutrition_dataset (Nutrition5k video frames) | 19,734 | only dishes provably in the official Nutrition5k train split (matched by exact nutrition signature), ≤16 frames per dish |
aryachakraborty/Food_Calorie_Dataset | 266 | human-estimated labels, down-weighted (0.4) |
Nutrition labels are taken directly from the official Nutrition5k dish metadata. Leakage controls, all enforced programmatically before training:
physical_meal_id); no dish appears in two splits.import torch, torch.nn as nn, torch.nn.functional as F
from transformers import AutoImageProcessor, SiglipVisionModel
from huggingface_hub import hf_hub_download
from PIL import Image
class NutritionHead(nn.Module):
def __init__(self, input_dim: int = 768):
super().__init__()
self.net = nn.Sequential(
nn.LayerNorm(input_dim), nn.Linear(input_dim, 768), nn.GELU(), nn.Dropout(.15),
nn.Linear(768, 256), nn.GELU(), nn.Dropout(.10), nn.Linear(256, 5))
def forward(self, x): return F.softplus(self.net(x))
MODEL_ID = "google/siglip2-base-patch16-224"
processor = AutoImageProcessor.from_pretrained(MODEL_ID)
vision = SiglipVisionModel.from_pretrained(MODEL_ID)
head = NutritionHead()
state = torch.load(hf_hub_download("Dralkh/CalorieK", "caloriek.pt"), map_location="cpu", weights_only=False)
vision.load_state_dict(state["vision_state"]); head.load_state_dict(state["head_state"])
vision.eval(); head.eval()
image = Image.open("meal.jpg").convert("RGB")
pixels = processor(images=image, return_tensors="pt").pixel_values
with torch.inference_mode():
out = vision(pixel_values=pixels)
feats = out.pooler_output if out.pooler_output is not None else out.last_hidden_state[:, 0]
kcal, mass_g, protein_g, carb_g, fat_g = head(feats.float())[0].tolist()
print(f"{kcal:.0f} kcal ± 119 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/C{carb_g:.0f}/F{fat_g:.0f}")
@misc{caloriek2026,
title={CalorieK: Accurate Food Calorie Estimation from a Single RGB Image},
author={Faisal Alkharji},
year={2026},
url={https://huggingface.co/Dralkh/CalorieK}
}