smolbge-image-embedding

SmolBGE: direct image embeddings for retrieval

Model weights · Method · Reproduce

SmolBGE converts an image directly into a 384-dimensional vector in BGE text space, without generating a caption. It lets image collections be searched with text queries and used as the image-retrieval component of a multimodal RAG system.

image → frozen Qwen3-VL image encoder → trained MLP → 384-d vector
text  → frozen BGE encoder                       → 384-d vector → cosine retrieval

Use

Python 3.12; sufficient accelerator memory is recommended. The first call downloads the model from Hugging Face.

pip install -r requirements.txt
python example.py --model yifanouyang/smolbge-image-embedding \
  --images photo.jpg --query "a person riding a bicycle"

from model import ImageEmbeddingModel

model = ImageEmbeddingModel.from_pretrained()
image_vectors = model.encode_images(["photo.jpg"])  # [1, 384]
query_vector = model.encode_text("a person riding a bicycle")
scores = image_vectors @ query_vector

Measured results

Text→image Recall@1 on the same custom candidate sets:

Training used 4,000 COCO + 6,000 Flickr8k images and 49,994 descriptions. DOCCI was used for evaluation only; a separate 500-image DOCCI confirmation set yielded 63.2% text→image Recall@1. Evaluation details · Split fingerprints · Data attribution.

Warmed batch-1 image processing on the same 96 images over three repeats:

SmolBGE improves retrieval accuracy over the SmolVLM adapter but takes longer to encode an image. The speed advantage applies only to the caption→BGE pipeline; caption quality was not evaluated on these retrieval sets. All times describe one local run, not a portable speed guarantee. Timing protocol and records.

Training code · Reproduction. Code and adapter weights are Apache-2.0; base-model and data terms remain separate. OCR, mixed indexes, multilingual queries and RAG answer quality remain untested.