clip-text-msmarco-corrector

clip-text-msmarco-corrector

A vec2text corrector model for the CLIP ViT-L/14 text encoder embeddings (the text encoder used by Stable Diffusion v1.5, identical to openai/clip-vit-large-patch14).

Given an initial text hypothesis from the inversion model, this model iteratively corrects it to better match the target embedding. It is used together with the inversion model Afrostnova/clip-text-msmarco-inversion — it is not used standalone and it reuses the inversion model's tokenizer.

• Base architecture: vec2text CorrectorEncoderModel
• Embedder: CLIPTextModel — openai/clip-vit-large-patch14
• Training data: MS MARCO
• Built on inversion alias: ClipTextModel-msmarco__msl32__58epoch

⚠️ Requirements

Requires the vec2text fork that supports the CLIPTextModel embedder (the one this model was trained with), not the upstream pip install vec2text package.

Usage

import vec2text

inv = vec2text.models.InversionModel.from_pretrained("Afrostnova/clip-text-msmarco-inversion")
cor = vec2text.models.CorrectorEncoderModel.from_pretrained("Afrostnova/clip-text-msmarco-corrector")
corrector = vec2text.load_corrector(inv, cor)

text = vec2text.invert_embeddings(
    embeddings=embeddings,   # CLIP text-encoder last_hidden_state pooled at EOS
    corrector=corrector,
    num_steps=20,
    sequence_beam_width=1,
)