qwen3-vl-4b-gui-agent

Fine-tuned Qwen3-VL-4B for GUI Click Actions

This model is a fine-tuned version of Qwen/Qwen3-VL-4B-Instruct trained on GUI trajectory data for click action prediction.

Training Details

• Base Model: Qwen/Qwen3-VL-4B-Instruct
• Training Checkpoint: Step 137, Epoch 0
• Task: Predict click coordinates from screenshot + instruction

Usage

With Transformers

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from PIL import Image

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "BLR2/qwen3-vl-4b-gui-agent",
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("BLR2/qwen3-vl-4b-gui-agent")

# Load your screenshot
image = Image.open("screenshot.png")
instruction = "Click on the search button"

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": instruction},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, 
    tokenize=True, 
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=50)
response = processor.decode(outputs[0], skip_special_tokens=True)
print(response)  # Outputs coordinates like "0.5234 0.7891"

With vLLM

vllm serve BLR2/qwen3-vl-4b-gui-agent --dtype bfloat16

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

# For vLLM with vision, encode image as base64
response = client.chat.completions.create(
    model="BLR2/qwen3-vl-4b-gui-agent",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "data:image/png;base64,{base64_image}"}},
                {"type": "text", "text": "Click on the search button"}
            ]
        }
    ],
    max_tokens=50
)
print(response.choices[0].message.content)

Output Format

The model outputs normalized coordinates in the format: x y where both values are in range [0, 1].

To convert to pixel coordinates:

x_norm, y_norm = map(float, output.split())
x_pixel = int(x_norm * image_width)
y_pixel = int(y_norm * image_height)