vqwen-qformer-tiktok-v2

Visual-Qwen — TikTok Sludge Detector

A trimodal sludge-content classifier for short-form videos: BLIP-2 (EVA-CLIP-G/14 + Q-Former) + a Linear projector (frozen) + Qwen3-4B with LoRA, augmented with Whisper V3 Turbo audio transcripts. Identifies "sludge" — short videos that stack multiple unrelated visual feeds on one screen, often paired with mismatched audio, engineered to bypass moderation by exploiting unimodal classifiers.

Performance

Held-out 300-video Kaggle stratified test split (video-level majority vote across 4 sampled frames):

Confusion matrix (TN=117, FP=8, FN=2, TP=173). 95% CIs from 1,000-iteration bootstrap resampling.

Additional out-of-distribution evaluation on 262 fresh TikToks (scraped May 2026, labelled by Gemma-4-31B-IT): 92.37% accuracy (95% CI 89.31-95.42). See the dataset ood-data/ folder for the raw eval set.

Architecture

input frame  ─► EVA-CLIP-G/14 vision encoder (frozen, 1B params)
                          │
                          ▼
              Q-Former + 32 query tokens (frozen)
                          │
                          ▼
              Linear projector (frozen, 4M params)  ◄── audio transcript (Whisper V3 Turbo)
                          │                               │
                          └───────────┬───────────────────┘
                                      ▼
                  Qwen3-4B (4B params) + LoRA r=16 (~32M trainable)
                                      │
                                      ▼
                         {sludge | non-sludge} + explanation

The Linear projector is trained in a stage-1 image-caption pre-training pass and then frozen during LoRA fine-tuning on the sludge corpus. Freezing the projector outperformed training it by +0.77 pp on the same eval — see paper section 4.

Usage

import torch
from transformers import Blip2ForConditionalGeneration, Blip2Processor

MODEL_ID = "alpharomercoma/vqwen-qformer-tiktok-v2"
model = Blip2ForConditionalGeneration.from_pretrained(
    MODEL_ID, dtype=torch.bfloat16, device_map="auto",
)
processor = Blip2Processor.from_pretrained(MODEL_ID)

# Single frame + ASR transcript -> sludge / non-sludge
from PIL import Image
frame = Image.open("frame.jpg").convert("RGB")
transcript = "subway surfers gameplay over family guy clip"  # from Whisper V3 Turbo

messages = [{"role": "user", "content": [
    {"type": "text", "text": f"Audio transcript: {transcript}"},
    {"type": "image"},
    {"type": "text", "text": "Is this sludge content? Answer yes or no."},
]}]
prompt = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=frame, return_tensors="pt").to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Run video-level by sampling 4 frames uniformly across the clip and majority-voting the predictions.

Training data

• 2,000 TikTok and YouTube Shorts videos (1,163 sludge / 837 non-sludge after correction).
• Hand-labelled, then label-refined via a Qwen3-VL-30B-A3B teacher + Gemma-3-27B-IT cross-judge audit pass (231 video flips, 11.55% noise rate in the original GT).
• Multi-task supervision: 25% classify, 15% layout, 20% describe, 35% coupled (classify+explain), 5% honest-refusal templates.

Dataset available on Kaggle — corrected labels, live OOD eval set, and raw mp4s in one dataset.

Code & paper

• Code: github.com/alpharomercoma/vqwen-qformer
• Project page: alpharomercoma.com/thesis
• Live demo: Hugging Face Space

Citation

@inproceedings{coma2026visualqwen,
  title={Visual-Qwen: Augmenting Multimodal Deep Learning with Attention Mechanisms for Sludge Content Detection},
  author={Coma, Alpha Romer and others},
  year={2026},
  organization={FEU Institute of Technology}
}