clip-qt-aft

QT-AFT: Quality Text-guided Adversarially Fine-Tuned CLIP

Adversarially robust CLIP vision-encoder checkpoints from:

Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models Futa Waseda, Saku Sugawara, Isao Echizen — ACM Multimedia (ACM MM) 2025 arXiv:2507.16257 | ACM DL

Summary

Quality Text-guided Adversarial Fine-Tuning (QT-AFT) improves the zero-shot adversarial robustness of CLIP by using high-quality image captions to guide adversarial examples during fine-tuning, instead of relying on short class-label texts (as in prior supervised adversarial training) or no textual guidance at all (as in unsupervised approaches). Guiding perturbations with rich captions steers them away from the diverse semantics present in natural images, producing vision encoders with substantially improved zero-shot robustness across 16 datasets.

Files

Each file is a full training checkpoint (a dict with keys such as epoch, step, state_dict, add_prompter, vision_encoder_state_dict, best_acc1, optimizer). The fine-tuned vision encoder weights are stored under vision_encoder_state_dict and load directly into model.visual of the corresponding OpenAI CLIP model. The encoder was adversarially fine-tuned from the original OpenAI CLIP visual encoder using PGD attacks (eps=4/255, 10 steps), while an image-to-text contrastive loss aligned adversarial image embeddings with captions generated by InternVL2.5-8B for ImageNet-1k images. The text encoder is unmodified from the original OpenAI CLIP checkpoint.

Usage

Requires the openai/CLIP package (pip install git+https://github.com/openai/CLIP.git).

import clip
import torch
from huggingface_hub import hf_hub_download

repo_id = "futakw/clip-qt-aft"
ckpt_path = hf_hub_download(repo_id=repo_id, filename="CLIP-ViT-B-16_QTAFT_eps4.pt")

model, preprocess = clip.load("ViT-B/16", device="cpu", jit=False)
# weights_only=False is required (the checkpoint stores full training state, not just tensors)
checkpoint = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.visual.load_state_dict(checkpoint["vision_encoder_state_dict"], strict=True)

For CLIP-ViT-L-14_QTAFT_eps4.pt, load the base model with clip.load("ViT-L/14", ...) instead.

Training data

ImageNet-1k images paired with detailed captions generated by InternVL2.5-8B (prompt: "Describe the image in detail within 50 words.").

Citation

@inproceedings{waseda2025qtaft,
  title     = {Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models},
  author    = {Waseda, Futa and Sugawara, Saku and Echizen, Isao},
  booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia (ACM MM)},
  year      = {2025},
  doi       = {10.1145/3746027.3755623}
}

License

MIT, consistent with the original OpenAI CLIP release. Base CLIP weights © OpenAI.