comic-panel-vlm-v1

Comic Panel VLM Embedder v1

Comic Panel VLM Embedder v1 is a multimodal panel feature extraction model designed to produce rich 512-dimensional embeddings for individual comic book panels.

It represents Stage 3 of the Comic Analysis Framework v2.0, and is trained downstream of CoSMo v4 (the page classifier that filters raw archives to narrative content). Its embeddings are intended as input to Stage 4 sequence modeling and similarity search over comic page collections.

This v1 iteration uses VLM-enriched panel descriptions generated by Gemini 2.5 Flash Lite — replacing the sparse OCR text used in prior versions — enabling the model to ground panel embeddings in narrative content (character descriptions, dialogue, mood, scene context) rather than raw detected text alone.

---

Model Architecture

The model is based on the PanelFeatureExtractor class. It fuses three independent modalities per panel into a single 512-dim embedding using an adaptive gated fusion mechanism.

1. Visual Encoder — Dual Backbone (~111M params, frozen)

Both visual backbones are frozen during training. Their 512-dim features are combined using a learned attention fusion: a 2-layer MLP computes a softmax weight over the two streams, producing a single 512-dim visual feature.

2. Text Encoder (~22M params, frozen)

Panel text is constructed from the VLM analysis JSON: description + joined text_content[].text dialogue. The backbone is frozen; a single linear projection maps 384 → 512.

3. Compositional Encoder (~0.2M params, trainable)

A 3-layer MLP encodes 7 spatial/layout features per panel:

Panel bounding boxes are sourced from the VLM JSON box_2d field ([y1, x1, y2, x2] in 0–1000 normalised coordinates, converted from Gemini 2.5 Flash Lite output).

4. Adaptive Fusion (~0.8M params, trainable)

An AdaptiveFusion module independently normalises each modality with LayerNorm(512), then computes a 3-way softmax gate over the concatenation of all three features plus optional modality presence indicators. The final embedding is a weighted sum of the three normalised modalities plus a small learned residual.

Total: ~115M params | ~2.5M trainable (frozen backbones)

---

Training

Training Objectives

Loss = 1.0 × L_contrastive + 0.5 × L_reconstruction + 0.3 × L_modality_alignment

• Contrastive: Panels from the same page should be mutually similar (temperature=0.07)
• Reconstruction: Predict one masked panel embedding from the remaining context
• Modality alignment: Cross-entropy alignment between vision and text embeddings for the same panel

Loss Curve

---

Input Format

The model operates per-panel. For a given page it expects:

• Panel image crop: (3, 224, 224) float32 tensor, normalised with ImageNet mean/std
• Panel text tokens: input_ids + attention_mask from all-MiniLM-L6-v2 tokenizer (max 128 tokens), constructed from description + dialogue
• Compositional features: (7,) float32 tensor of spatial/layout values
• Modality mask: (3,) binary indicator — [has_vision, has_text, has_comp]

At the page level, up to 16 panels are batched together with zero-padding. A boolean panel_mask (N,) indicates valid vs padded slots.

---

Output Format

panel_embeddings: (N_panels, 512)  # float32, one embedding per panel

When run over a full dataset via generate_stage3_embeddings_vlm.py, output is stored in a Zarr store:

stage3_embeddings_vlm.zarr/
├── panel_embeddings   shape: (N_pages, 16, 512)  float32
└── panel_masks        shape: (N_pages, 16)        bool

---

Usage

Because the model requires VLM-annotated panel JSONs, inference uses the pipeline scripts from the Comic Analysis Repository.

Quick Start — Load Model

import torch
from stage3_panel_features_framework import PanelFeatureExtractor

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = PanelFeatureExtractor(
    visual_backbone='both',
    visual_fusion='attention',
    feature_dim=512,
    freeze_backbones=True
).to(device)

checkpoint = torch.load('best_model_vlm.pt', map_location=device)
state_dict = checkpoint.get('model_state_dict', checkpoint)
model.load_state_dict(state_dict)
model.eval()

Generate Embeddings for a Dataset

python src/version2/generate_stage3_embeddings_vlm.py \
  --manifest manifests/master_manifest_20251229.csv \
  --vlm_cache_dir /data/vlm_cache \
  --pss_labels pss_labels_v1.json \
  --checkpoint checkpoints/stage3_vlm/best_model_vlm.pt \
  --output_zarr stage3_embeddings_vlm.zarr \
  --output_metadata stage3_metadata_vlm.json \
  --batch_size 64 \
  --num_workers 8

---

Intended Use

This model is designed as an intermediate representation layer in a comic analysis pipeline:

1. CoSMo v4 classifies pages → filters to narrative pages
2. This model embeds panels → 512-dim per-panel features
3. Stage 4 (PanelSequenceTransformer) contextualises panel sequences → strip embeddings
4. Stage 5 search performs similarity search over the final embeddings

Panel embeddings from this model are suitable for:

• Similarity search over individual panels (find visually/narratively similar panels)
• Input to sequence models that require panel-level features
• Downstream clustering or classification of panels

They are not recommended for cover/advertisement pages — the model was trained exclusively on narrative story pages and its embedding space reflects that distribution.

---

Citation

If you use this model or the Comic Analysis Framework, please reference the repository:

@misc{comic-analysis-framework,
  author = {RichardScottOZ},
  title  = {Comic Analysis Framework v2.0},
  year   = {2026},
  url    = {https://github.com/RichardScottOZ/Comic-Analysis}
}