UIPro-7B_Stage1

UIPro: Unleashing Superior Interaction Capability For GUI Agents

<div align="center">

[💻Code] [🚀Quick Start] [📝Paper]

</div>

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Brave Group, CASIA
• Model type: Vision-Language Model
• Language(s) (NLP): English
• License: Apache License 2.0
• Finetuned from model [optional]: Qwen2-VL-7B-Instruct

Model Sources [optional]

UIPro_1stage-7B is a GUI grounding model finetuned from Qwen2-VL-7B-Instruct.

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/ZJULiHongxin/UIPro
• Paper [optional]: https://arxiv.org/abs/2509.17328

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

First, ensure that the necessary dependencies are installed:

pip install transformers
pip install qwen-vl-utils

Inference code example:

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
            },

            # For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
            {"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
            # For AutoGUI
            {"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
            # For VisualWebBench Element Grounding
            {"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
        ],
    }
]

Evaluation

It is recommended to use AutoGUI evaluation suite based on LMMS-EVAL to evaluate it on multiple GUI Grounding benchmarks.

Results

Comparison on the GUI element grounding benchmarks. UIPro achieves impressive grounding accuracy, especially on FuncPred, RefExp, and VWB EG. AnyRes means using an image division strategy to handle images with variable resolutions.

References: [1] Qwen2VL [2] CogAgent [3] SeeClick [4] Ferret-UI [5] UGround [6] OS-ATLAS-Base

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@InProceedings{Li_2025_ICCV,
    author    = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
    title     = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {1613-1623}
}

Framework versions

• PEFT 0.11.1