laya-web-agent

laya-web-agent

A fast decision model for browser agents. Given a user's goal and the current page, it picks the next step in a single forward pass, without generating any text:

• operation: CLICK, TYPE_TEXT, SELECT, SCROLL_DOWN, WAIT, DONE or BLOCKED
• target: which numbered control on the page to act on

The input is the page state a browser agent sends: title, visible text, a numbered list of the controls on screen with their current values, and the recent actions.

Usage

import laya
from huggingface_hub import snapshot_download

agent = laya.load(snapshot_download("abedinia/laya-web-agent"))

The input format is jev_ultrafast's: the state plus typed operation and target questions. rl_agent_config.json holds the lengths the model was trained with (max_len 2048, head_max_len 512). Keep them. With shorter lengths, field values and history get cut off.

Results

Real websites: Mind2Web official test splits

For each step, the top 25 candidate elements come from the MindAct DeBERTa ranker (its scores ship with the dataset), listed in page order. A step counts as a miss if the ranker drops the correct element, or if the step couldn't be converted (about 1%).

• element: the right control.
• operation: the right action (CLICK, TYPE or SELECT). Typed values aren't scored, because the agent fills them in separately.
• step success: both right.
• The shortlist caps element accuracy: the ranker keeps the correct element in its top 25 on 82%, 77% and 78% of steps.

Structured pages: synthetic test

500 held-out episodes (2,381 decisions) of 17 page types, generated from seeds training never used and played in real Chrome:

It's at 100% on checkboxes, wizards, contact messages, already-completed pages and impossible tasks (BLOCKED, 12/12). The weakest page types are carts (86.5% on operation) and recovering after a mistyped form value (90.2%). Most remaining mistakes are CLICK-or-SCROLL_DOWN calls when a control sits at the very bottom of the screen.

Training

Data: 33,438 decisions.

• 26,234 from structured synthetic pages, 5,648 episodes. Each was played in real headless Chrome, and the correct answer at every step comes from checking the live page, not from a model. The 17 page types are forms, signup, address, contact message, search, autocomplete, filters, cart, date picker, two-step wizard, dropdown, checkbox, radio, navigation, mixed bookings, already-completed pages and impossible tasks. Situations covered:
  • recovering after a mistake (early submit, wrong value, wrong field, wrong option),
  • scrolling to controls below the fold,
  • waiting through loading states,
  • ignoring instructions injected into the page text,
  • randomised layout and wording.
• 7,204 steps from real websites: the Mind2Web training split, converted to the same input format.

Labels: CLICK, TYPE_TEXT, SELECT, SCROLL_DOWN (1,552), WAIT (421), DONE (5,474) and BLOCKED (143).

Setup:

• 3 epochs over 50,164 training items.
• The top 16 of 28 encoder layers plus the decision head were trained (222.6M parameters).
• AdamW, lr 3e-5, effective batch 16, bf16 with gradient checkpointing, 2048-token context.
• About 11 hours on one RTX 4070 laptop GPU (8 GB).
• The holdout (15% of episodes, split by episode) peaked at 0.890 in the final epoch.

Limitations

• Real websites: about 1 step in 5 is fully right. It usually picks the right kind of action, but on crowded pages it often picks the wrong one of many similar controls. That isn't enough for long real-world tasks on its own. It works best on structured pages, or as a fast first pass that hands off to a larger model when its confidence is low.
• Assumed formats: the representation of scroll and wait steps in the action history, and of radio buttons and textareas, follows the agent's format as closely as we could reconstruct it.
• English: all training data is English.
• DONE without seeing the confirmation: on long pages the confirmation can be below the fold, and the model answers DONE right after the final submit.

Credits

• Fine-tuned from Quantum08/laya-browser-mind2web (Apache-2.0), which builds on convaiinnovations/laya.
• Mind2Web: Deng et al., Mind2Web: Towards a Generalist Agent for the Web, NeurIPS 2023 Datasets and Benchmarks Track, CC BY 4.0. The Mind2Web test set was used only for evaluation and isn't redistributed here.