GUI-G2-3B

GUI-G2-3B

This repository contains the GUI-G2-3B model from the paper GUI-G²: Gaussian Reward Modeling for GUI Grounding. We provided more inference details on the github quick start. We will update GUI-G2-3B results on GUI Grounding benchmark.

Model Description

The model is based on Qwen2.5-VL-3B-Instruct and is fine-tuned using our proposed Gaussian dense reward framework framework.

• 💡Gaussian Point & Coverage Rewards: Encourage accurate, spatially-aligned clicks.

• 📏 Adaptive Variance Mechanism: Adjusts reward granularity based on element scale.
• 🌍 Dense Learning Signals: Smooth gradients outperform binary RL rewards in early-stage learning.
• 📊 State-of-the-art Performance on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro datasets.

Quick Start

First, install the required dependencies:

pip install transformers==4.49.0 qwen-vl-utils

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
     "inclusionAI/GUI-G2-3B",
     torch_dtype=torch.bfloat16,
     attn_implementation="flash_attention_2",
     device_map="auto")
     
processor = AutoProcessor.from_pretrained("inclusionAI/GUI-G2-3B")
image_path = ''
instruction = ''

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "image_path",
            },
            {"type": "text", "text": instruction},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

📊 Results on ScreenSpot-v2

---

🙏 Acknowledgement

The RL Training code build from VLM-R1 project.

📄 Citation

If you use GUI-G², please cite our work:

@misc{tang2025guig2gaussianrewardmodeling,
      title={GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding}, 
      author={Fei Tang and Zhangxuan Gu and Zhengxi Lu and Xuyang Liu and Shuheng Shen and Changhua Meng and Wen Wang and Wenqi Zhang and Yongliang Shen and Weiming Lu and Jun Xiao and Yueting Zhuang},
      year={2025},
      eprint={2507.15846},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.15846}, 
}