videomae-qwen-connectors

VideoMAE-Qwen Connector Weights

Trained connector layers for fusing VideoMAE-v2 with Qwen2.5-VL for video captioning.

Contents

This repository contains only the trained connector weights (Perceiver Resampler + MLP Projector), not the full model.

• Perceiver Resampler: Compresses 1568 VideoMAE tokens to 64 query tokens
• MLP Projector: Projects 768-dim features to 2048-dim Qwen embeddings

Base models (frozen during training):

• VideoMAE: AnnaelleMyriam/videomaev2-finetuned-finebio
• Qwen: Qwen/Qwen2.5-VL-3B-Instruct

Code

# Create model
model = VideoMAEQwenFusion(
    videomae_model_path="AnnaelleMyriam/videomaev2-finetuned-finebio",
    qwen_model_path="Qwen/Qwen2.5-VL-3B-Instruct",
    num_query_tokens=64,
    resampler_num_heads=8,
    resampler_num_layers=2,
    mlp_hidden_dim=4096,
    mlp_num_layers=3,
    mlp_dropout=0.1,
)

# Download connector weights
weights_path = hf_hub_download(
    repo_id="AnnaelleMyriam/videomae-qwen-connectors",
    filename="connector_weights.pth"
)

# Load weights
connector_weights = torch.load(weights_path, map_location="cpu")
model.load_state_dict(connector_weights, strict=False)

model.eval()
model.to("cuda")

License

Apache 2.0