freezeshift-dual-vlm

FreezeShift Dual — Bounded Encoder Adaptation

A bounded-adaptation image-text retrieval model: 49.2M deployed parameters, of which 4.9M are trainable.

Research artifact. Not licensed for commercial use. See NOTICE.

Code: https://github.com/Pranav210901/Efficient-VLM

What this is

A frozen DINOv3 ViT-S/16 vision tower (224 px) and a frozen all-MiniLM-L6-v2 text tower, joined by learned token aggregation and residual projections into a shared normalised 384-d space. MobileCLIP2-S0 provided a distillation signal during training only and is not part of this model.

Identical to M_T1 plus rank-128 LoRA on the attention projections of the final four blocks of both towers. This is the study's upper bound on what limited encoder adaptation recovers: +9.3pp test R@1 over the fully frozen model for 2.0M extra trainable parameters.

Results

Flickr30k Karpathy test, opened once after the model was frozen, with no training of any kind. This checkpoint (seed 44) scores 62.04% mean bidirectional R@1.

Three-seed means for context:

This model does not beat the compact references. MobileCLIP2-S0 is smaller than OpenCLIP and scores higher than both models released here. The contribution is a measured boundary on frozen-encoder alignment under a fixed latency budget, not a state-of-the-art retrieval model.

Zero-shot classification transfer is weak (CIFAR-100 ~36-38%, Oxford-IIIT Pet ~8-10%, EuroSAT ~22-24%) and should be treated as a negative result.

Seed selection

The seed was chosen on the validation split, never on test. Selecting on test would have picked a different seed and inflated the reported figure.

Checkpoint provenance

This exports the COCO-dev-selected epoch. The headline validation figures in the dissertation (54.487% / 63.416%) use Flickr-validation epoch selection; under the deployed selection the same models score 54.36% / 63.04% on validation. The test figures above are unaffected by this distinction.

Files

The adapter-only variant contains no third-party weights, which avoids redistributing DINOv3 entirely if that matters for your use.

Usage

# requires the repo code on PYTHONPATH
from load_model import load

model = load(".", adapter_only=False)          # or True for the adapter variant
image_embeds = model.encode_image(pixel_values) # (B, 384), L2-normalised
text_embeds  = model.encode_text(["a dog on a beach"])

Licensing

The DINOv3 License requires that derivatives be distributed under its terms with a copy of the Agreement provided, that publications acknowledge DINO Materials, and that use comply with Trade Controls. It prohibits military, weapons, espionage, and nuclear applications. MobileCLIP2-S0 weights are not redistributed here.

Citation

Pokhrel, P. Efficient Frozen Vision-Language Alignment. MSc dissertation, University of Surrey, 2026.