sam3.1-int8-int4-convrot

SAM 3.1 Multiplex — Native ConvRot INT8 and INT4

Smaller, native-ComfyUI quantizations of Meta's SAM 3.1 Multiplex checkpoint. Both variants use ConvRot weight and activation quantization for selected transformer linears while keeping CLIP and quality-sensitive layers in FP16. They load through ComfyUI's standard checkpoint loader—no custom quantization node or loader is required.

• INT8 W8A8: the conservative default; 1.19 GiB and very close to FP16 in our spot checks.
• INT4 W4A4: the smallest release; 0.98 GiB, with surprisingly similar masks in our testing.
• FP16 CLIP: included in both files for stock-ComfyUI compatibility.
• Selective quantization: embeddings, normalization, convolutions, geometry and mask heads, final projections, and other sensitive or low-value operations remain FP16.

SAM 3.1's headline change is Object Multiplex for faster multi-object video tracking. Meta's release notes focus on optimized tracking and do not claim an image-detector accuracy upgrade, so this should not be read as “SAM 3.0, but smarter at still-image segmentation.” See the official SAM 3.1 release notes.

Downloads

Relative to Meta's FP32 source, the INT8 file is 63.42% smaller and the INT4 file is 70.08% smaller. INT4 is 18.19% smaller than INT8.

ComfyUI v0.27.0 introduced native INT8 ConvRot support. ComfyUI v0.28.0 added native INT4 ConvRot support. INT4 matrix multiplication is used on supported GPUs; ComfyUI may fall back to INT8 multiplication on unsupported hardware. Keep ComfyUI and comfy-kitchen current, especially on Turing or AMD hardware where backend fixes have continued to land.

Installation and native workflow

Put either .safetensors file in:

ComfyUI/models/checkpoints/

Then use the normal native chain:

Load Checkpoint → CLIP Text Encode → SAM3 Detect

The files may also live in any checkpoint directory configured through extra_model_paths.yaml. Select the checkpoint with the standard Load Checkpoint node. Do not use a legacy INT8 custom loader.

These releases were validated on the native image-segmentation path. The checkpoint also contains SAM 3.1 Multiplex tracking weights, but video tracking has not received the same breadth of release testing yet.

Quantization recipe

The conversion starts from Meta's official FP32 sam3.1_multiplex.pt, not from the FP16 repack. Its 3,502,755,717-byte size and SHA-256 match the file served by facebook/sam3.1:

0567DEBEEC80BA4AC6369540C6C248025283CB3FF2B92827509E57E2B3541CB6

For each of 230 eligible block-linear weights, the converter:

1. upcasts the source weight to FP32;
2. applies a block-Hadamard ConvRot transform;
3. chooses an MSE-optimized clipping scale;
4. stores per-row quantization scales and native ComfyUI quantization metadata.

The INT8 file uses W8A8 operations: 8-bit weights with online 8-bit activation quantization at those linears. The INT4 file similarly uses W4A4 operations. Unselected floating-point tensors are stored as FP16. This is dynamic per-operation activation quantization, so no calibration image set is embedded in the files.

Performance

Quantization improves file size reliably. Speed is more workload- and GPU-dependent because SAM's preprocessing, mask refinement, convolutions, CPU/GPU synchronization, and relatively small matrix multiplications can dominate the quantized linears.

Controlled warm benchmark on an RTX 3090, PyTorch 2.11 + CUDA 13.0, comfy-kitchen 0.2.22, 1280×720 input, six detected people, threshold 0.3, and two native refinement passes. Values are the median of four model-resident runs:

With refinement disabled, a CUDA operator profile showed INT8 reducing total self GPU time by approximately 17.5%, while time in the main eligible linear operations fell by approximately 32%. In ordinary single-object workflows, warm FP16 and INT8 measurements were both commonly around 0.6–0.7 seconds; a measurable end-to-end speedup is therefore not guaranteed. Image size, detection count, refinement count, GPU generation, offloading, and model warm-up all matter.

Quality spot checks

On the pictured official Meta truck.jpg demo image, using the prompt truck, threshold 0.3, and two refinement passes:

On a separate six-person test with refinement disabled, union-mask IoU versus FP16 was 0.99882 for INT8 and 0.98498 for INT4. These are useful regression and visual checks, not a substitute for evaluation across a segmentation benchmark. Some prompts or difficult edges may regress even when these examples look nearly identical.

Why these two variants?

The release candidates were deliberately narrowed to selective quantization with FP16 CLIP:

• Quantized CLIP is not yet broadly native-compatible. It saves another approximately 226 MB, but ComfyUI's SAM 3.1 text-encoder state-dict conversion must also remap the quantization companion tensors. The experimental files work with a local ComfyUI patch, not an unmodified public release, so distributing them now would create a fragile support burden.
• Aggressive targeting offers almost nothing here. In our candidates it saved only about 14 MB beyond selective quantization, produced no repeatable speed gain, and increases the chance that a sensitive projection or short-matrix operation causes a quality regression.
• FP16 CLIP is cheap during repeated segmentation. Text encoding is a small portion of an image run and can be reused, while native compatibility is much more valuable.
• Selective INT4 already held up well. A mixed INT4/INT8 checkpoint remains possible, but initial tests did not justify its added complexity or larger file size.

Future improvements

• Upstream-compatible quantized CLIP: extend ComfyUI's SAM text-encoder key conversion to include ConvRot configuration and scale tensors, add round-trip tests, and then repeat prompt-quality evaluation. This is the clearest remaining checkpoint-size reduction.
• Batched and cached refinements: the current native image node refines detections serially and can recompute the 1008-pixel vision path for the same crop across iterations. Batching crops and caching image features should improve real workflows more than quantizing another handful of small layers. This requires native-node code changes, not a new checkpoint.
• SAM-specific fusion or kernels: ConvRot already runs through specialized comfy-kitchen kernels. Larger gains would likely require fusing SAM-specific preprocessing, detector, and refinement operations, reducing dispatch and synchronization, or compiling stable subgraphs. That is substantial, hardware-sensitive engineering rather than a straightforward weight conversion.
• Broader activation quantization: activations are already A8/A4 at quantized linears. Extending low-precision activations into convolutions, geometry processing, or mask heads would require representative calibration, backend support, and careful accuracy analysis; the initial release avoids that risk.
• Video validation: benchmark long clips, object-count scaling, memory use, and tracking stability across FP16, INT8, and INT4. SAM 3.1's largest upstream gains are in this path.
• Wider hardware coverage: publish reproducible results for Turing, Ampere, Ada, Blackwell, AMD/ROCm, and low-VRAM offload configurations.

Checksums

D4AF12DF7DB2881ED44476DEBF555A5ACE15234572E6C25EBAA3AF3183AD2133  sam3.1_multiplex_int8_convrot_selective_fp16clip_fp32source.safetensors
A75D0559F508C30FC418FD060A5E3F330F1B7085ED2636F5FD67E3384A2EBBBD  sam3.1_multiplex_w4a4_convrot_selective_fp16clip_fp32source.safetensors

License and credits

The original checkpoint is from Meta's gated SAM 3.1 repository. The gate controls access to the upstream download; it does not replace the license. The included weights are derivatives of the SAM Materials and remain subject to the SAM License, which permits derivative works and redistribution only under that same agreement and requires a copy of the agreement to accompany the files. SAM_LICENSE.txt is included for that purpose. Review the full terms before uploading or using the checkpoints.

The Comfy-Org SAM 3.1 repository uses the same SAM License for its repackaged FP16 weights. ComfyUI's GPL-3.0 software license applies to ComfyUI code; it does not relicense Meta's model weights.

Credits:

• Meta SAM 3 / SAM 3.1 for the model, source weights, and demo image.
• Comfy-Org/ComfyUI and comfy-kitchen for native SAM 3.1 and ConvRot execution support.

The comparison and promo use Meta's official assets/images/truck.jpg. The promo's abstract background was generated specifically for this release.