Qwen3-VL-4B-SAMTok-dam

Qwen3-VL-4B-SAMTok-dam

Dedicated mask-understanding checkpoint for region-conditioned text generation. This repository belongs to SAMTok: Representing Any Mask with Two Words (CVPR 2026), a unified mask-token interface for multimodal language models.

Paper · Code · Collection · Project page

Model overview

SAMTok keeps the backbone's vision-language architecture and represents each region by two learned mask-code tokens. A separate image-conditioned mask tokenizer converts between these tokens and a 2D mask. Mask generation predicts tokens and decodes them against the image; mask understanding first encodes a supplied region and inserts its tokens into the user instruction. Loading only the language model produces textual tokens, not PNG masks.

Mask-token format

<|mt_start|><|mt_0230|><|mt_0345|><|mt_end|>

There are 512 code tokens: 0000–0255 represent the first codebook and 0256–0511 represent the second. Subtract 256 from the second token index before calling the mask decoder. In the example above, decoder codes are [230, 89]. The two boundary tokens delimit one mask; the two code tokens carry its region representation. The codes are image-conditioned: the same pair is not a globally fixed object category or a mask independent of the input image.

Use the tokenizer/processor shipped with this checkpoint. Do not add the mask tokens again or replace it with the unmodified backbone tokenizer. Preserve mask tokens when decoding output; skip_special_tokens=False is a robust choice for downstream parsing.

This checkpoint is intended for region captioning and understanding. Use a -co or recommended release when you need text-to-mask generation.

Evaluation results

Scores below are transcribed from the supplied SAMTok v2 paper (arXiv:2601.16093v2). Table numbers refer to that PDF. These are reported results, not a new benchmark run. Each table applies to this checkpoint and training setting only.

Region captioning — Tables 7–9

MDVP-Bench and VideoRefer-D are zero-shot evaluations. VideoRefer-D uses a single frame, not native video mask reconstruction. SC: subject correspondence; AD: appearance description; TD: temporal description; HD: hallucination detection. Preserve each benchmark’s native score scale. The model-zoo note uses different table numbers; the numbers above follow the supplied PDF.

Quickstart

1. Environment and source code

Use a Linux CUDA environment with a compatible PyTorch/torchvision pair. The example loads the VLM and SAMTok on one GPU. Leave memory for both models, image tokens and the generation cache; the nominal model size is not the total GPU-memory requirement. The paper uses A100 80 GB, but does not report minimum inference VRAM per checkpoint.

git clone https://github.com/bytedance/Sa2VA.git
cd Sa2VA
# Install a CUDA-compatible PyTorch/torchvision pair first.
python -m pip install 'transformers==4.57.1' 'huggingface_hub>=0.26,<1' \
  accelerate qwen-vl-utils pillow numpy timm einops hydra-core \
  pycocotools mmengine peft xtuner
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
git rev-parse HEAD
python -m pip freeze > samtok_environment.txt
python -c "from transformers import AutoModelForImageTextToText; from projects.samtok.models import VQ_SAM2, VQ_SAM2Config, SAM2Config, DirectResize; print('Imports OK')"

The project package imports training wrappers eagerly, so mmengine, peft and xtuner can be required even for inference. The optional original visualization demo additionally imports Detectron2; the inference.py example here saves binary PNG masks directly and does not use that visualizer. The suggested Transformers version supports the relevant backbone families, but this package combination has not been runtime-certified with all released checkpoints. Resolve any upstream dependency constraints and retain the final environment file with evaluation results.

2. Download this checkpoint

export SAMTOK_REPO='zhouyik/Qwen3-VL-4B-SAMTok-dam'
export SAMTOK_DIR="$PWD/checkpoints/Qwen3-VL-4B-SAMTok-dam"
hf download "$SAMTOK_REPO" --local-dir "$SAMTOK_DIR"

The inference assets are the VLM weights and configuration, its processor/tokenizer/chat template, mask_tokenizer_256x2.pth (the trained SAMTok parameters), and sam2.1_hiera_large.pt (SAM 2.1 Large initialization). They are different files with different roles. Confirm both auxiliary files exist before running. If an experimental checkpoint omits an auxiliary file, obtain the matching shared SAMTok asset from the recommended 8B repository, verify compatibility, and pass its actual local path explicitly; do not use a random quantizer or a differently sized codebook.

test -f "$SAMTOK_DIR/mask_tokenizer_256x2.pth"
test -f "$SAMTOK_DIR/sam2.1_hiera_large.pt"

For a frozen experiment, first obtain HfApi().model_info(repo_id).sha, then use hf download --revision <that-SHA>. Record the auxiliary assets' revisions as well.

3. Run inference

Copy inference.py from the downloaded checkpoint into the Sa2VA root (cp "$SAMTOK_DIR/inference.py" ./inference.py). It loads the correct backbone through AutoModelForImageTextToText, builds SAMTok with explicit 256 × 2 configuration, preserves raw generated tokens, decodes each mask, resizes to the original image dimensions, and saves results. It uses BF16 for the VLM and the loaded SAMTok module's own dtype for mask processing. The published PLM configuration declares FP32; BF16 in this example is a memory-saving inference choice, not a claim about the exact paper evaluation dtype.

PYTHONPATH=. python inference.py \
  --model "$SAMTOK_DIR" \
  --sam2 "$SAMTOK_DIR/sam2.1_hiera_large.pt" \
  --mask-tokenizer "$SAMTOK_DIR/mask_tokenizer_256x2.pth" \
  --image projects/samtok/figs/totoro.jpg \
  --task region \
  --mask /absolute/path/to/binary_region.png \
  --output "outputs/Qwen3-VL-4B-SAMTok-dam"

For a user image, replace --image with its path. For a region mask, foreground is any nonzero pixel and its dimensions must match the original RGB image exactly. Choose a nonempty mask. The script writes response.txt, codes.json, and mask_000.png, mask_001.png, etc. for generated masks. For segmentation it also writes union.png, including an all-zero mask when the model predicts no target. Region-caption inference produces text and does not save a predicted segmentation mask.

For mask generation, use --task segment --phrase "your referring expression". For interleaved caption generation, use --task gcg. For global-image region captioning, use --task region --mask region.png. Use --task gres-rl specifically for the GRES RL checkpoint. These examples demonstrate the shared interface; task-specific fine-tuning does not establish benchmark performance on every other task.

4. Understand the processing contract

The VLM image processor and the SAMTok image processor have separate roles. Keep the checkpoint's VLM processing defaults. SAMTok uses DirectResize(1024) on the original RGB uint8 image, converts to CHW, and casts to the SAMTok dtype/device. Do not insert an additional division by 255 or an unrelated image normalization into that path. Decode with forward_with_codes, bilinearly resize to (original_height, original_width), then apply the released threshold > 0.5. For generalized segmentation, union the per-instance masks before scoring. For GCG, preserve each mask separately and align it to the associated phrase.

For input regions, compute the binary mask's bounding box in the original image, normalize it by [width, height, width, height], and call the tokenizer encoder with the image, mask and normalized box. Add the second-codebook offset when formatting the resulting pair as text. The original image and region must remain spatially aligned; a crop requires a correspondingly cropped mask and re-encoding.

Reproducing the paper evaluation

Use the exact checkpoint named above, its tokenizer/processor, the trained mask tokenizer, and the original benchmark annotations and splits. Record the Sa2VA Git commit, the Hugging Face checkpoint SHA, dataset version, package versions, GPU model, prompts, generation settings and evaluation logs. The paper reports NVIDIA A100 80 GB GPUs. It does not publish a complete version-locked inference environment; the installation in Quickstart is a starting environment, not a certified reproduction lockfile.

The single-image example is a smoke test. It rejects malformed mask groups so errors are visible. The official evaluators have their own repair rules (including a missing-second-code fallback of -1), task-specific output parsing, and metric computation. Use those rules for comparing to paper scores; do not substitute the demo parser or a generic IoU script.

For DLC-Bench, download godx7/DLC-Bench to ./godx7/DLC-Bench; the script expects DLC-bench.json and images/ there. Use qwen3vl_dam_infer.py. This script also uses args.sam2_path without declaring it: add parser.add_argument('--sam2_path', required=True) immediately before args = parser.parse_args() if your checkout still has that omission.

hf download godx7/DLC-Bench --repo-type dataset --local-dir godx7/DLC-Bench
PYTHONPATH=. python projects/samtok/evaluation/qwen3vl/qwen3vl_dam_infer.py \
  --model_path "$SAMTOK_DIR" \
  --vq_sam2_path "$SAMTOK_DIR/mask_tokenizer_256x2.pth" \
  --sam2_path "$SAMTOK_DIR/sam2.1_hiera_large.pt"

The benchmark inference applies zoom-in preprocessing when the region bounding-box area is below 30% of image area: it expands small box dimensions to at least 140 pixels, clips to the image, and conditionally resizes the crop, then encodes both the global and cropped masks. Retain the script’s crop/image pairing and exact prompt. The single-image region example below uses the global image only; it is not equivalent to this benchmark preprocessing.

Use qwen3vl_mdvp_infer.py and qwen3vl_videorefer_d_single_frame.py for MDVP-Bench and single-frame VideoRefer-D. Their dataset paths and benchmark-specific scorers must be configured from the corresponding benchmark releases. Region-caption evaluation can depend on an external judge, its version and prompts; inference output alone is not the final benchmark score. A fully pinned judge configuration and verified scoring command are not established by the sources audited for this card.

Training details

The shared SAMTok tokenizer is initialized from SAM 2.1 Large and trained on 209 million masks. It freezes the SAM image and prompt encoders while training the mask decoder and residual quantizer. The default uses two non-shared 256-entry codebooks, AdamW, global batch size 1024, initial learning rate 4e-5, and cosine decay.

The paper's mask-task VLM SFT corpus contains approximately five million conversations covering generation, region understanding and interactive tasks. Mask annotations are pre-tokenized and training uses next-token prediction. For the Qwen VLM experiments, the image encoder is frozen while the projector and language model are fine-tuned; the paper reports AdamW, global batch size 256 and learning rate 2e-5 with cosine decay. The recommended release additionally mixes general VQA data according to the model zoo. Exact mixture proportions and all per-checkpoint training durations are not specified in the inspected release documentation.

The paper describes Easy-R1 based GRPO at learning rate 1e-6, 26k reasoning cold-start samples, and challenging subsets of 8k GRES / 41k GCG samples. This describes the RL variants; it does not mean every checkpoint underwent RL. See training instructions and the RL implementation for the released recipes.

Limitations and troubleshooting

• Missing model class or import error: use a Transformers build that supports the checkpoint family and the repository's helper APIs. Resolve the Sa2VA training-wrapper dependencies before loading weights. A syntax check alone does not validate this environment.
• Embedding size mismatch / no mask tokens: use the processor from this exact SAMTok checkpoint. The base backbone tokenizer lacks the learned vocabulary extension.
• Auxiliary checkpoint not found: Hugging Face repo IDs are not filesystem paths inside SAM2Config. Download the files and pass real local paths.
• Incorrect masks: check the second-codebook offset, the image-mask alignment, the 256 × 2 non-shared configuration, and the original-image output size. Do not change preprocessing or the threshold while comparing paper results.
• Malformed or truncated output: preserve response.txt, check generation length, and use the official benchmark's parsing/repair policy. The example fails explicitly on malformed groups instead of silently inventing codes.
• Out of memory: reduce concurrent samples or image-token load for an exploratory run. Any change to image resolution, dtype, quantization or generation limits should be recorded and validated before a paper comparison.
• Unexpectedly high or unchanged metrics: isolate prediction caches by checkpoint and split; confirm all intended sample IDs are evaluated and no stale files are included.

SAMTok is a learned, lossy image-conditioned mask representation. Errors can come from either token prediction or mask reconstruction, particularly for small, thin, occluded or ambiguous regions. The released tokenizer represents 2D image masks; single-frame VideoRefer-D evaluation does not establish temporal mask tracking or native video-mask reconstruction. Task-specific post-training can change behavior on tasks outside its target benchmark, and general backbone scores are not SAMTok checkpoint scores.

License and attribution

Use is subject to the licenses and terms of the released checkpoint, its backbone, SAM 2.1, and the source code. This card does not assign a new license to the model weights. A license identifier has not been inferred from an unrelated backbone or repository; preserve any existing repository license metadata when publishing. Dataset licenses and benchmark access conditions remain separate.

Citation

@article{samtok,
  title={SAMTok: Representing Any Mask with Two Words},
  author={Zhou, Yikang and Zhang, Tao and Gong, Dengxian and Wu, Yuanzheng and Tian, Ye and Wang, Haochen and Yuan, Haobo and Wang, Jiacong and Qi, Lu and Fei, Hao and Wang, Anran and Wang, Zhuochen and Wang, Yujing and Chen, Cheng and Ji, Shunping and Li, Xiangtai},
  journal={CVPR},
  year={2026}
}

Sources and validation status

Checkpoint identity and training variant follow the official model zoo; numerical targets and table numbering follow the author-supplied samtok_v2.pdf (arXiv v2). Inference follows the released generation demo and the linked task evaluators. This card and its accompanying example were statically reviewed; full checkpoint inference and benchmark reproduction have not been executed in the preparation environment. Exact benchmark parity therefore remains to be validated with the datasets, complete scoring dependencies and original evaluation settings.