laya-mind2web-browser-agent

Laya Fine-Tuned for Browser Automation (Mind2Web)

This is a fine-tuned version of the Laya decision model, specialized for browser automation tasks. It predicts the next browser action (operation and target element) from a web page's DOM structure and a user goal.

Model Details

• Developed by: Shaun Andrade (@ShaunSpark)
• Base model: convaiinnovations/laya (421M parameters)
• Training data: Mind2Web (671 usable training examples)
• Validation accuracy: 74.3% on a held-out set of 68 examples

Training Procedure

• Hardware: Google Colab, single NVIDIA T4 GPU (16 GB VRAM)
• Optimizer: 8-bit AdamW (bitsandbytes), learning rate 1e-5
• Effective batch size: 8 (micro-batch 1, gradient accumulation 8)
• Precision: FP16 mixed precision (torch.cuda.amp)
• Memory optimizations: Gradient checkpointing, expandable_segments
• Epochs: 2 (best checkpoint; further training overfits)

Intended Use

Decision engine for a browser automation agent. Given a goal and a list of interactive DOM elements, the model answers two questions in one forward pass:

• Operation: CLICK, TYPE, or SELECT
• Target: the correct element index from the candidate list

Limitations

• Trained on a single dataset (Mind2Web); may not generalize to all websites.
• Overfits beyond 2 epochs on 671 examples.
• Requires the base Laya tokenizer and architecture.

How to Use

import torch
import laya

agent = laya.load("convaiinnovations/laya")
model = agent.model
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

Citation

• Laya: https://huggingface.co/convaiinnovations/laya
• Mind2Web: https://arxiv.org/abs/2306.06070