fc-clip

FC-CLIP — Open-Vocabulary Panoptic Segmentation

FC-CLIP is an open-vocabulary panoptic segmentation model that pairs a frozen ConvNeXt-Large CLIP backbone with a lightweight Mask2Former decoder. It achieves strong zero-shot performance without requiring separate specialist models for things vs. stuff.

This repository hosts the COCO Panoptic checkpoint uploaded to HuggingFace Hub by Claude, for testing use with the FiftyOne Model Zoo.

Attribution

Paper: "A Simple Framework for Open-Vocabulary Segmentation and Detection"
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.
CVPR 2023 · arxiv 2311.15539

Original code: bytedance/fc-clip — Apache 2.0 license

Usage

Standalone (trust_remote_code)

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("neerajaabhyankar/fc-clip", trust_remote_code=True)
model.eval()

# Preprocess: RGB uint8 numpy/PIL → normalised tensor
pixel_values = model.preprocess_image(your_pil_image)  # [1, 3, H, W]

with torch.no_grad():
    results = model(pixel_values)

panoptic_seg, segments_info = results[0]
# panoptic_seg: int32 tensor [H, W]  — pixel → segment id
# segments_info: list[{"id", "category_id", "isthing"}]

Open-vocabulary (custom classes)

results = model(pixel_values, class_names=["cat", "dog", "sky", "grass"])

Architecture

Requirements

torch torchvision transformers open_clip_torch safetensors

No detectron2 required — the model is self-contained.

License

Apache 2.0 (same as original bytedance/fc-clip)