Downloads · 30 days
0
llamma07/clip-vit-l14-film-still-lora
clip-vit-l14-film-still-lora is a machine learning model from llamma07. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
A LoRA fine-tune of OpenAI's CLIP ViT-L/14, adapted to retrieve film stills from natural-language cinematographic descriptions (setting, lighting, palette, framing, atmosphere).
Downloads · 30 days
0
Access
Public
Updated Jul 26, 2026
Repo size
1.1 MB
Likes
1
Public
Click a slice to open those files.
.pt1.1 MB · 99%
From the Hugging Face model README
A LoRA fine-tune of OpenAI's CLIP ViT-L/14, adapted to retrieve film stills from natural-language cinematographic descriptions (setting, lighting, palette, framing, atmosphere).
out_proj layer of each nn.MultiheadAttention block, in both the visual and text transformer towersgemma4:latest (9.6 GB). The exact quantization/precision of this tag was not separately verified - it reflects whatever Ollama serves under :latest for this model as of training time, which may vary across pulls.Recall@k on a held-out test set of 1,097 (image, caption) pairs, retrieval performed against the full test set:
| Metric | Base CLIP ViT-L/14 | Fine-tuned (this model) |
|---|---|---|
| R@1 | 36.8% | 76.3% |
| R@5 | 59.6% | 93.2% |
| R@10 | 68.4% | 96.6% |
This is a custom LoRA implementation, not a standard PEFT adapter - the LoRA delta is merged directly into the out_proj.weight property, since nn.MultiheadAttention reads .weight/.bias directly rather than calling the module's forward(). You need the LoRALayer class below to load it correctly.
import torch
import torch.nn as nn
import clip
RANK = 4
ALPHA = 32
class LoRALayer(nn.Module):
def __init__(self, original_layer, rank=RANK, alpha=ALPHA):
super().__init__()
self.original = original_layer
d_in = original_layer.weight.shape[1]
d_out = original_layer.weight.shape[0]
self.lora_A = nn.Parameter(torch.randn(rank, d_in) * 0.02)
self.lora_B = nn.Parameter(torch.randn(d_out, rank) * 0.02)
self.scale = alpha / rank
@property
def weight(self):
return self.original.weight + (self.lora_B @ self.lora_A) * self.scale
@property
def bias(self):
return self.original.bias
def forward(self, x):
return nn.functional.linear(x, self.weight, self.bias)
def apply_lora(model, rank=RANK, alpha=ALPHA):
for name, module in model.named_modules():
if isinstance(module, nn.MultiheadAttention):
if hasattr(module, 'out_proj'):
setattr(module, 'out_proj', LoRALayer(module.out_proj, rank=rank, alpha=alpha))
return model
# load base CLIP
model, preprocess = clip.load("ViT-L/14")
model = model.float()
model = apply_lora(model, rank=RANK, alpha=ALPHA)
# load the LoRA weights from this repo
lora_weights = torch.load("lora_best.pt", map_location="cpu")
model.load_state_dict(lora_weights, strict=False)
model.eval()
Once loaded, use model.encode_image(...) and model.encode_text(...) exactly as you would with standard CLIP - the LoRA adaptation is transparent at inference time.
Retrieving visually/tonally similar film stills given a descriptive text query (e.g. "wide shot of a snowy mountain pass at dusk, cold blue tones, lone figure, isolation"). Best suited to descriptive, cinematographic-style queries similar to the training caption style, rather than short generic labels.
Training and evaluation were run on a MacBook Air with Apple M4 chip, 16 GB unified memory - but on CPU only. The training and evaluation scripts explicitly set device = "cpu" and do not use Apple's MPS/Metal backend, despite running on M-series hardware capable of it. Anyone reproducing this should expect CPU-only training times accordingly; switching device to "mps" in the scripts would likely speed this up substantially but was not what was used for the reported results.
Caption generation (via Gemma/Ollama) was performed locally on the same machine.
The images used to train this adapter were frames extracted from copyrighted films; they are not included in this repository or distributed alongside these weights - only the trained LoRA parameters (numeric deltas, not images) are shared here. Whether training on copyrighted stills and releasing the resulting fine-tuned weights raises copyright considerations is an unsettled legal question at the time of this release, and this model card makes no claim about the legal status of downstream use. Users evaluating this model for commercial or redistributive purposes should make their own assessment, and are encouraged to consult their own legal counsel.