CLIPSeg-rd64-refined-ExecuTorch

CLIPSeg (rd64-refined) — ExecuTorch (segmentation from a text prompt)

Say what you want and get a mask for it. The shelf's other open-vocabulary model, Grounding DINO, returns boxes for a phrase; SAM and EdgeTAM return a mask for a click. This returns a mask for a phrase, which neither of those does.

Two files, because the expensive half does not depend on the prompt:

clipseg_vision   pixel_values (1, 3, 352, 352) fp32
    -> three activations, each (1, 485, 768)     # CLIP ViT-B/16 layers 3, 6, 9
clipseg_segment  a3, a6, a9, input_ids (1, 77) int64
    -> logits (1, 352, 352) fp32                 # per-pixel; sigmoid for 0..1

• Source: CIDAS/clipseg-rd64-refined — 150.7M parameters
• License: apache-2.0
• Threshold: logits > 0 is the mask. Everything below is background.

Run the vision half once per image and the segment half once per phrase. That is the whole point of the split: trying "a tree", "the sky" and "grass" on one photograph costs one ViT pass and three cheap ones, not three ViT passes. The three activations are 4.5 MB in total — an intermediate the caller holds, not a file.

Checked against upstream's single-shot forward: max_abs_diff 0.000e+00 on every prompt. Splitting it changes nothing about the arithmetic.

Variants

*Mac arm64, median of 10 — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine is 53 ms for the vision half and 19 ms for the segment half.

The vision half is slower than eager on XNNPACK and there is no Core ML build for it. Both are worth knowing before you plan around this model:

• Core ML refuses a graph holding any non-contiguous _clone_dim_order node (AssertionError: Only contiguous memory format is supported in CoreML). The vision half has two — ExecuTorch's dim-order pass puts the patch convolution's output in channels-last, and interpolate_pos_encoding produces another. Rewriting the patch convolution as the equivalent linear was tried and only moved the problem: the reshape-and-permute that gathers patches earns a channels-last clone of its own. The segment half has no 4-D convolution and lowers to Core ML whole, 100% delegated.
• On XNNPACK the vision half delegates 66.8% of its ops across 43 subgraphs. Why it is 3.7x slower than eager has not been measured, so this card does not name a cause.

Verification

Correlation over a mask is not the gate. A mask gets used by thresholding it, so the number that matters is how many pixels change side:

Only prompts the reference actually answers are scored. On the gate image the model answers "a tree" (63% of pixels), "the sky" (21%) and "grass" (5%), and ignores "a road" (0%) — an empty mask agrees with another empty mask perfectly, so scoring the ignored prompt would flatter every build. The four prompts were picked by running the reference over a dozen candidates rather than written from imagination.

Running it

1. Preprocess to 352x352, with CLIPSegProcessor. The CLIP backbone was trained at 224 and CLIPSeg runs it at 352 by interpolating the position embeddings — feeding it 224 gives a mask at the wrong scale rather than an error.

2. Tokenise the prompt to a fixed 77, CLIP's context length:

ids = processor.tokenizer([prompt], padding="max_length", truncation=True,
                          max_length=77, return_tensors="pt")["input_ids"]

Padding to the full 77 is safe here and it is worth knowing why: CLIP's text encoder is causal and pools at the end-of-text token, so the padding that follows never reaches it. Measured against the processor's own tight padding: 6.676e-06.

3. Vision once, segment per prompt, then threshold at 0 and resize the 352x352 mask back to your image.

Not shipped: int8

Both halves are withheld together, because the gate measures the pair. clipseg_vision would come out at 83.5 MB from 286.7 (dynamic int8 has almost nothing to leave alone in a ViT — there is no token embedding table), and clipseg_segment at 141.6 MB, which is larger than its own fp16 at 129.6 MB: the segment half carries CLIP's 49,408 x 512 text embedding table, 101 MB of its 260.8, a 38.8% share. This shelf's rule int8/fp16 = 0.5 + 1.5 x (table share) predicts 1.082 and it came out at 1.092.

The quality is what decides it. Worst mask IoU 0.9429 against a 0.95 bar, and the worst case is the smallest mask — "grass", 5% of the image, where boundary pixels are most of the mask. Correlation for the same build reads 0.9967 and 0.9999 on the two halves, which no correlation gate would stop.

This repo holds both towers: clip_vit_b32_image_xnnpack_fp32.pte (image) and clip_vit_b32_text_xnnpack_fp32.pte (text, fixed len 77 + attention mask). L2-normalize both embeddings, then cosine-match.