mt1-frozen-vlm

M_T1 — Fully Frozen Vision-Language Alignment

A fully frozen-backbone image-text retrieval model: 47.2M deployed parameters, of which 2.9M are trainable.

Research artifact. Not licensed for commercial use. See NOTICE.

Code: https://github.com/Pranav210901/Efficient-VLM

What this is

A frozen DINOv3 ViT-S/16 vision tower (224 px) and a frozen all-MiniLM-L6-v2 text tower, joined by learned token aggregation and residual projections into a shared normalised 384-d space. MobileCLIP2-S0 provided a distillation signal during training only and is not part of this model.

Both backbones are completely frozen. The trainable set is a two-block 256-d learned vision-token aggregator (C4), a 128-d four-head learned-query text aggregator, two residual projection heads, and the logit scale.

Results

Flickr30k Karpathy test, opened once after the model was frozen, with no training of any kind. This checkpoint (seed 43) scores 51.90% mean bidirectional R@1.

Three-seed means for context:

This model does not beat the compact references. MobileCLIP2-S0 is smaller than OpenCLIP and scores higher than both models released here. The contribution is a measured boundary on frozen-encoder alignment under a fixed latency budget, not a state-of-the-art retrieval model.

Zero-shot classification transfer is weak (CIFAR-100 ~36-38%, Oxford-IIIT Pet ~8-10%, EuroSAT ~22-24%) and should be treated as a negative result.

Seed selection

The seed was chosen on the validation split, never on test. Selecting on test would have picked a different seed and inflated the reported figure.

Checkpoint provenance

This exports the COCO-dev-selected epoch. The headline validation figures in the dissertation (54.487% / 63.416%) use Flickr-validation epoch selection; under the deployed selection the same models score 54.36% / 63.04% on validation. The test figures above are unaffected by this distinction.

Files

The adapter-only variant contains no third-party weights, which avoids redistributing DINOv3 entirely if that matters for your use.

Usage

# requires the repo code on PYTHONPATH
from load_model import load

model = load(".", adapter_only=False)          # or True for the adapter variant
image_embeds = model.encode_image(pixel_values) # (B, 384), L2-normalised
text_embeds  = model.encode_text(["a dog on a beach"])

Licensing

The DINOv3 License requires that derivatives be distributed under its terms with a copy of the Agreement provided, that publications acknowledge DINO Materials, and that use comply with Trade Controls. It prohibits military, weapons, espionage, and nuclear applications. MobileCLIP2-S0 weights are not redistributed here.

Citation

Pokhrel, P. Efficient Frozen Vision-Language Alignment. MSc dissertation, University of Surrey, 2026.