rf-detr-mobile-gui-detection

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #e2e8f0; border-radius: 8px; box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;"> <div style="background: #1e40af; padding: 20px; color: white;"> <h1 style="margin: 0; font-size: 22px; font-weight: 700; color: white; border: none;">rf-detr-mobile-gui-detection</h1> <p style="margin: 8px 0 0 0; font-size: 13px; color: #bfdbfe; font-weight: 500;">Mobile GUI grounding model built on top of roboflow/rf-detr-medium</p> </div> <div style="display: flex; gap: 8px; flex-wrap: wrap; padding: 12px 20px; background: #f8fafc; border-bottom: 1px solid #e2e8f0;"> <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Object Detection</span> <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">DETR</span> <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Mobile GUI</span> <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Grounding</span> </div> <div style="padding: 20px; display: flex; flex-direction: column; gap: 16px;"> <p style="margin: 0; font-size: 14px; color: #334155; line-height: 1.6;"><b>rf-detr-mobile-gui-detection</b> is a mobile gui grounding model built on top of <b>roboflow/rf-detr-medium</b> using the <b>rfdetrforobjectdetection</b> architecture. rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone, an rf-detr windowing pattern for efficient attention, a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.</p> <div style="border: 1px solid #fde68a; padding: 14px; border-radius: 4px; background: #fffbeb;"> <span style="font-weight: 700; color: #92400e; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">Note</span> <p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.5;">rf-detr: neural architecture search for real-time detection transformers: <a href="https://huggingface.co/papers/2511.09554" style="color: #92400e; text-decoration: underline;">https://huggingface.co/papers/2511.09554</a></p> </div> </div> </div>

Metrics Loss Map

Per Class Metrics

Quick Start with Transformers

pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install torchvision==0.23.0 transformers==5.9.0 accelerate gradio==6.19.0

import gradio as gr
import torch
from PIL import Image, ImageDraw

from transformers import AutoImageProcessor, RfDetrForObjectDetection

# Load model and processor
model_name = "prithivMLmods/rf-detr-mobile-gui-detection"

processor = AutoImageProcessor.from_pretrained(model_name)
model = RfDetrForObjectDetection.from_pretrained(model_name)

# Detection threshold
THRESHOLD = 0.35


def detect_gui(image):
    image = Image.fromarray(image).convert("RGB")

    inputs = processor(images=image, return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    target_sizes = torch.tensor([image.size[::-1]])
    results = processor.post_process_object_detection(
        outputs,
        target_sizes=target_sizes,
        threshold=THRESHOLD,
    )[0]

    draw = ImageDraw.Draw(image)

    detections = []

    for score, label, box in zip(
        results["scores"],
        results["labels"],
        results["boxes"],
    ):
        box = [round(x, 2) for x in box.tolist()]
        label_name = model.config.id2label[label.item()]
        confidence = round(score.item(), 3)

        # Draw bounding box
        draw.rectangle(box, outline="red", width=3)

        # Draw label
        draw.text(
            (box[0] + 4, max(0, box[1] - 16)),
            f"{label_name} {confidence:.2f}",
            fill="red",
        )

        detections.append(
            {
                "Label": label_name,
                "Confidence": confidence,
                "Bounding Box": box,
            }
        )

    return image, detections


demo = gr.Interface(
    fn=detect_gui,
    inputs=gr.Image(type="numpy", label="Upload Mobile UI Screenshot"),
    outputs=[
        gr.Image(type="pil", label="Detected GUI Elements"),
        gr.JSON(label="Detections"),
    ],
    title="RF-DETR Mobile GUI Detection",
    description="Upload a mobile UI screenshot to detect GUI elements using RF-DETR.",
)

if __name__ == "__main__":
    demo.launch()

e.g., demo screenshot

Acknowledgements

• roboflow/rf-detr-medium: rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone (with an rf-detr windowing pattern for efficient attention), a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.

• mobile ui design detection[dataset] by mrtoy: this dataset is designed for object detection tasks focused on detecting elements in mobile ui designs. the target objects include text, images, and groups. the dataset contains mobile ui images with object detection bounding boxes, class labels, and localization information.