PhoneUIAnchor-829M

PhoneUIAnchor-829M

PhoneUIAnchor-829M is a vision-language model for locating elements in mobile and web interfaces. Given a screenshot and a natural-language description, it returns the normalized center point of the target element. It can be used with GUI agents, test automation, accessibility tools, and similar visual workflows.

This repository contains a complete, standalone checkpoint that can be loaded directly with Transformers without additional model files.

Capabilities

• Intent grounding: locate the control required to complete an action.
• Description grounding: locate an element from visible or semantic traits.
• Function grounding: locate a control from a detailed functionality description.
• Cross-resolution output: return normalized 0-999 coordinates that can be mapped to any screenshot size.
• Local deployment: run from a single 829M-parameter model without an external inference API.

Technical Profile

Installation

pip install -r requirements.txt

Usage

The packaged Florence-2 implementation uses custom model code. Review the included Python files and load with trust_remote_code=True.

import re
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "lumimate/PhoneUIAnchor-829M"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).cuda().eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

image = Image.open("ui_screenshot.png").convert("RGB")
prompt = (
    'Where is the "Settings" element? '
    '(Output the center coordinates of the target)'
)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
    "cuda", dtype=torch.bfloat16
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=16,
    )

text = processor.tokenizer.batch_decode(
    output_ids, skip_special_tokens=False
)[0]
match = re.search(r"<loc_(\d+)>,<loc_(\d+)>", text)
point = tuple(map(int, match.groups())) if match else None

x_px = point[0] / 999 * image.width
y_px = point[1] / 999 * image.height
print({"normalized": point, "pixels": (x_px, y_px)})

A command-line implementation is included in inference_example.py.

Intended Use

PhoneUIAnchor-829M is intended for research and product prototyping in GUI perception, agentic interaction, automated testing, and assistive interfaces. Predictions should be validated before high-impact or irreversible automation.

Limitations

• Development data emphasizes mobile user interfaces and may not cover every desktop, game, embedded, or highly customized UI style.
• Coordinate quality can degrade on very small, occluded, or visually ambiguous controls.

Architecture and Licensing

PhoneUIAnchor-829M uses the Florence-2 architecture and is distributed under the terms provided in LICENSE.