MedSAM-Agent-Qwen3-VL-8B-MedSAM2-GGUF

MedSAM-Agent-Qwen3-VL-8B-MedSAM2 (GGUF)

GGUF quantizations of Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2, produced with Unsloth + llama.cpp. These run with llama.cpp (multimodal / mtmd) and tools built on it (e.g. Ollama).

• Base (source): Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2 (~17 GB BF16)
• Quantization: Q4_K_M (K-quant, ~4.85 bit/weight, good size/quality trade-off)
• MMProj: the vision-projector weights are distributed separately as *_mmproj.gguf (BF16), as is standard for VLM GGUF.

What this model is

MedSAM-Agent reformulates interactive medical image segmentation into a multi-step, autonomous, decision-making process. It drives a segmentation tool (SAM-family) using MLLM-generated tool calls (add_bbox, add_point, stop_action). This repository is the GGUF-quantized Qwen3-VL-8B grounding model without the separate MedSAM2 segmentation checkpoint (that lives in the code repo).

See the source repo for the full description, running scripts, and citation.

Run with llama.cpp

./llama.cpp/build/bin/llama-mtmd-cli \
  -m MedSAM-Agent-Qwen3-VL-8B-MedSAM2.Q4_K_M.gguf \
  --mmproj MedSAM-Agent-Qwen3-VL-8B-MedSAM2.BF16-mmproj.gguf \
  --jinja --ctx-size 8192
# load an image inside the runner:
#   /image path/to/image.png

Model files

Citation

Please cite the original work:

@misc{liu2026medsamagentempoweringinteractivemedical,
      title={MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning},
      author={Shengyuan Liu and Liuxin Bao and Qi Yang and Wanting Geng and Boyun Zheng and Chenxin Li and Wenting Chen and Houwen Peng and Yixuan Yuan},
      year={2026},
      eprint={2602.03320},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.03320},
}