slugvision-500m

slugvision-500m

A 500M-parameter vision-language model that turns an uploaded image (plus an optional article title) into a 3–5 word kebab-case permalink slug, e.g. three-brown-teddy-bears or nursing-workforce-growth-scaling. Built for cheap CPU serving: ~1.6s per image cold (process start + model load + inference) on 6 desktop threads via llama.cpp.

It is a LoRA fine-tune (r16, merged) of SmolVLM2-500M-Video-Instruct, distilled from a Gemma teacher on ~31k image→slug pairs spanning five slices of an editorial-image distribution: everyday photos, conceptual editorial illustrations, product shots, clinical stock imagery, and web screenshots. Training data is published at kelnei/slugvision.

A larger sibling, slugvision-2.2b, scores noticeably better on untitled photos at ~10x the CPU latency. The full pipeline (corpus generation, training, eval, export) is at github.com/kelnei/slugvision.

Input contract

The model was trained on exactly two prompt templates. Use them verbatim — other phrasings will degrade quality silently.

Without a title (slug describes visible content):

Create a URL slug for this image: 3 to 5 lowercase words joined by hyphens. Reply with the slug only.

With an article title (slug reflects the article topic; image steers emphasis):

This image appears in an article titled: "{title}"
Create a URL slug for the image in this article's context: 3 to 5 lowercase words joined by hyphens. Reply with the slug only.

Images were preprocessed with size={"longest_edge": 1024} during training; match it at inference (the bundled GGUF mmproj already embeds this cap).

Usage

llama.cpp (CPU serving)

llama-mtmd-cli -m slugvision-500m-q8_0.gguf --mmproj mmproj-f16.gguf \
  --image photo.jpg -n 32 --temp 0 \
  -p 'Create a URL slug for this image: 3 to 5 lowercase words joined by hyphens. Reply with the slug only.'

Transformers

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

processor = AutoProcessor.from_pretrained("kelnei/slugvision-500m", size={"longest_edge": 1024})
model = AutoModelForImageTextToText.from_pretrained(
    "kelnei/slugvision-500m", torch_dtype=torch.bfloat16
)

messages = [{"role": "user", "content": [
    {"type": "image", "url": "photo.jpg"},
    {"type": "text", "text": "Create a URL slug for this image: 3 to 5 lowercase words joined by hyphens. Reply with the slug only."},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Files

Evaluation

Blind pairwise LLM-judge against the teacher on a frozen 1,000-image holdout (rubric 1–5 per slug; the judge never knows which is which). Token-F1 is overlap with the teacher's slug.

Format validity is ~99–100% across slices. When a title is provided the model is near parity with its 26B-class teacher; on untitled photos the gap is real (capacity-bound — more data and higher LoRA rank did not close it; the 2.2B sibling does).

CPU latency (llama.cpp, 6 threads, full cold start per image): Q8_0 1.6s, Q4_K_M 1.7s; image encode dominates. Fits comfortably in a small serverless container (~0.8GB of weights, <2GB RAM).

Training

LoRA r16 on all linear layers, 1 epoch over 30,739 pairs, batch 16, bf16, images capped at 1024px longest edge. Labels authored by a Gemma-family teacher (26B-class, locally hosted); illustration/screenshot/half-of-clinical labels were generated with the source article headline in context, which is what trains the titled mode.

License and provenance

• Base model: SmolVLM2-500M-Video-Instruct, Apache-2.0.
• The fine-tuning labels are outputs of a Gemma-family model, so this distilled model is distributed under the Gemma Terms of Use with its prohibited use policy; hence the gemma license tag.
• Photo-slice training images came from COCO train2017 and are not redistributed here or in the dataset repo; all other training images are locally generated (FLUX.1-schnell renders, self-authored HTML screenshots) and ship with the dataset.

Limitations

• English slugs only.
• Tuned for editorial-image distributions (article cards); out-of-domain images (documents, memes, dense text) are untested.
• The slug is not guaranteed unique — collision handling belongs in your permalink generator.