GlimpsePrune-LLaVA-1.5-13B

GlimpsePrune: A Dynamic Visual Token Pruning Framework for Large Vision-Language Models

The model was presented in the paper A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models. Official code repository: https://github.com/HVision-NKU/GlimpsePrune

<div align="center"> <img src="https://github.com/HVision-NKU/GlimpsePrune/blob/main/assets/case1.png?raw=true" width="80%"> <img src="https://github.com/HVision-NKU/GlimpsePrune/blob/main/assets/case2.png?raw=true" width="80%"> <br> <em>GlimpsePrune dynamically prunes a large number of irrelevant visual tokens before answering questions, reducing the model's inference overhead.</em> </div>

GlimpsePrune is a dynamic visual token pruning framework designed for Large Vision-Language Models (LVLMs). It introduces a novel approach inspired by human cognition to efficiently process high-resolution inputs by pruning irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.6% of visual tokens while on average fully retaining the baseline performance on free-form VQA tasks. The reduced computational cost also enables more effective fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate. Our work paves a new way for building more powerful and efficient LVLMs.

Abstract

Visual token compression is critical for Large Vision-Language Models (LVLMs) to efficiently process high-resolution inputs. Existing methods that typically adopt fixed compression ratios cannot adapt to scenes of varying complexity, often causing imprecise pruning that discards informative visual tokens and results in degraded model performance. To address this issue, we introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.6% of visual tokens while on average fully retaining the baseline performance on free-form VQA tasks. The reduced computational cost also enables more effective fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate. Our work paves a new way for building more powerful and efficient LVLMs.

✨ Key Features

• High Pruning Rate: Prunes over 90% of visual tokens on average with almost no performance loss, effectively reducing computational and memory overhead.
• Robust Performance: Stable performance when processing high-resolution images and handling complex free-form VQA tasks.
• Lightweight Training: Only a few extra parameters (Glimpse token and VIP) need to be trained, completed in less than 1 hour on a single A100 GPU.
• Broad Compatibility: Supports single and multi-image inputs, is compatible with KV-Cache and Flash Attention 2, and provides a fair comparison benchmark with other mainstream visual compression methods.

🖼️ Framework Overview

The core idea of GlimpsePrune is to introduce a glimpse token and a lightweight Visual tokens Important Predictor (VIP) that can quickly identify and retain the visual regions most relevant to the text prompt, pruning the remaining redundant information.

<div align="center"> <img src="https://github.com/HVision-NKU/GlimpsePrune/blob/main/assets/framework.png?raw=true" width="70%"> </div>

📊 Performance Results

We evaluated GlimpsePrune on multiple VQA benchmarks. The results show that it achieves a high pruning rate while maintaining performance on par with the original model, outperforming other visual compression methods.

<p align="center"> <b>Free-form VQA Benchmarks</b><br> <img src="https://github.com/HVision-NKU/GlimpsePrune/blob/main/assets/freeform_results.png?raw=true" width="90%"> </p> <p align="center"> <b>Short-form VQA Benchmarks</b><br> <img src="https://github.com/HVision-NKU/GlimpsePrune/blob/main/assets/shortform_results.png?raw=true" width="90%"> </p>

Usage

You can use GlimpsePrune with the llava_gp, which is located in the GitHub repository.

from llava.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN
from llava.conversation import conv_templates
from llava_gp.mm_utils import (
    get_model_name_from_path,
    process_images,
    process_bboxes,
    tokenizer_image_token,
)
from llava_gp.model.builder import load_pretrained_model

# Load model and processor
base_model = "liuhaotian/llava-v1.5-13b"
new_modules_dir = "ashun989/GlimpsePrune_LLaVA-1.5-13B"
device = "cuda:0"
torch_dtype = torch.bfloat16
model_name = get_model_name_from_path(base_model)
llava_model_args = {
    "attn_implementation": "flash_attention_2",
    "torch_dtype": torch_dtype,
}
tokenizer, model, image_processor, max_length = load_pretrained_model(
    base_model, None, model_name, device_map=device, **llava_model_args)
model.load_new_modules(new_modules_dir)
model.eval()
conv_mode = "vicuna_v1"

# Prepare image and text inputs
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)
query = "What is in the image?"

# Format the conversation for the model
input_ids = []
images = []
image_sizes = []
grid_h = grid_w = model.get_vision_tower().num_patches_per_side
if model.config.mm_use_im_start_end:
    query = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + '\n' + query
else:
    query = DEFAULT_IMAGE_TOKEN + '\n' + query
conv = conv_templates[conv_mode].copy()
conv.append_message(conv.roles[0], query)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

image_tensor = process_images([image], image_processor, model.config)[0]
input_id = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt')
input_ids.append(input_id)
images.append(image_tensor)
image_sizes.append(image.size)

input_ids = torch.stack(input_ids, dim=0).to(device=device)
images = torch.stack(images, dim=0).to(device=device, dtype=torch_dtype)

# Generate response
generate_ids = model.generate(
            input_ids=input_ids,
            images=images,
            image_sizes=image_sizes,
            generation_config=generation_config,
            do_selection=True,  # Enable glimpse prune
        )
response = tokenizer.batch_decode(generate_ids, skip_special_tokens=True,
                                  clean_up_tokenization_spaces=True)[0].strip()
print(response)

Citation

If you find our work helpful, please consider citing our paper:

@misc{zeng2025glimpseprune,
      title={A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models}, 
      author={Quan-Sheng Zeng and Yunheng Li and Qilong Wang and Peng-Tao Jiang and Zuxuan Wu and Ming-Ming Cheng and Qibin Hou},
      year={2025},
      eprint={2508.01548},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.01548}, 
}