MedSAM-Agent-Qwen3-VL-8B-MedSAM2-4bit

MedSAM-Agent-Qwen3-VL-8B-MedSAM2 (4-bit NF4, Unsloth)

4-bit (NF4 / bitsandbytes) quantization of Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2, produced with Unsloth's Dynamic 4-bit.

• Base (source): Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2 (~17 GB BF16)
• Quantized size: ~6.1 GB (the LLM is 4-bit; the vision encoder is kept in BF16 by Unsloth's default, which preserves accuracy)
• Method: Unsloth Dynamic 4-bit NF4 (bitsandbytes-compatible). config.json carries a quantization_config so it loads with transformers + bitsandbytes out of the box.
• Model card / license / usage of the original apply. Please see the source repo for the full README, running scripts, and citation.

What this model is

MedSAM-Agent reformulates interactive medical image segmentation into a multi-step, autonomous, decision-making process. It drives a segmentation tool (SAM-family) using MLLM-generated tool calls (add_bbox, add_point, stop_action). This checkpoint is the trained Qwen3-VL-8B grounding model without the separate MedSAM2 segmentation checkpoint (that lives in the code repo as MedSAM2_latest.pt).

Load

import torch
from transformers import AutoModel, AutoProcessor, BitsAndBytesConfig

model_id = "Weidows/MedSAM-Agent-Qwen3-VL-8B-MedSAM2-4bit"

quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModel.from_pretrained(model_id, quantization_config=quant_config, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

If you use Unsloth, just pass load_in_4bit=True / FastVisionModel.from_pretrained(model_id, load_in_4bit=True).

Usage (agentic segmentation)

Follow the official inference flow from the MedSAM-Agent repo — pass this model folder as --model-path:

cd infer
python run_single_inference.py \
  --img-path infer/demo/BTCV-0-106_CT_abdomen.png \
  --target-description "right kidney in abdomen CT" \
  --model-path /path/to/this/model \
  --seg-checkpoint /path/to/MedSAM2_latest.pt \
  --seg-model medsam

Quantization (Unsloth)

from unsloth import FastVisionModel
model, tokenizer = FastVisionModel.from_pretrained(
    "Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2",
    max_seq_length=8192,
    load_in_4bit=True,
    device_map="auto",
)
model.save_pretrained("MedSAM-Agent-Qwen3-VL-8B-MedSAM2-4bit")
tokenizer.save_pretrained("MedSAM-Agent-Qwen3-VL-8B-MedSAM2-4bit")

Citation

Please cite the original work:

@misc{liu2026medsamagentempoweringinteractivemedical,
      title={MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning},
      author={Shengyuan Liu and Liuxin Bao and Qi Yang and Wanting Geng and Boyun Zheng and Chenxin Li and Wenting Chen and Houwen Peng and Yixuan Yuan},
      year={2026},
      eprint={2602.03320},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.03320},
}