multimodal-vision-agent-lora

<div align="center">

Multimodal Vision Agent — LoRA Adapter

QLoRA fine-tuned adapter for Qwen2.5-7B-Instruct that converts natural language desktop UI instructions into structured browser automation actions

</div>

---

Table of Contents

• Model Details
• Supported Actions
• Quick Start
  • Installation
  • Inference
  • Full Agent Integration
• Real-World Test Results
• v2 Improvements (vs v1)
• Training Details
  • Dataset
  • Quantization
  • LoRA Configuration
  • Training Results
• Full Project Structure
• Dependencies
• License
• Citation

---

Model Details

This adapter fine-tunes Qwen2.5-7B-Instruct using QLoRA (4-bit NF4 quantization) to produce structured UI actions (click, type, navigate, scroll, wait, done) from natural language instructions. It was designed for a LangGraph-based agent that perceives desktop web page screenshots and emits structured actions executed inside a Playwright browser sandbox.

Why Qwen2.5-7B-Instruct?

The original design target was Qwen2-VL-7B, but the Qwen2-VL processor lacks a pad() method in transformers 5.x, causing data collator failures during training. Qwen2.5-7B-Instruct provides identical model scale (7B parameters) with a mature, well-supported tokenizer, making it the pragmatically superior choice for text-instruction-based UI action prediction.

Architecture Overview

The agent framework operates as a LangGraph state machine with three nodes:

1. Perception Node — Captures a browser screenshot + DOM snapshot, compresses action history, and feeds everything to the VLM.
2. Action Node — Executes the predicted action in the Playwright browser sandbox (click, type, navigate, scroll, wait).
3. Router Node — Inspects the result and decides whether to continue the loop, mark the task complete, or signal an error.

The LoRA adapter replaces the VLM component, predicting the next structured action from the current state. The full framework is available on GitHub.

Model Card

---

Supported Actions

The model outputs structured JSON inside <action> tags. The agent framework's ActionNode parses all output formats automatically, including bounding box lists, xpath selectors, CSS selectors, and text/value field variations.

Output Format Details

The model can produce bounding boxes in two formats:

• List format (most common): "bbox": [x, y, width, height]
• Object format: "bbox": {"x": ..., "y": ..., "width": ..., "height": ...}

The model also supports element targeting via:

• XPath selectors: "xpath": "//input[@name='username']"
• CSS selectors: "selector": "a[href='/signup']" or "selector": "#login_field"

---

Quick Start

Installation

pip install torch transformers peft bitsandbytes accelerate sentencepiece

Inference

import torch
import json
import re
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# --- Step 1: Configure 4-bit quantization ---
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

# --- Step 2: Load base model with quantization ---
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

# --- Step 3: Load LoRA adapter ---
model = PeftModel.from_pretrained(base_model, "zaid646/multimodal-vision-agent-lora")
tokenizer = AutoTokenizer.from_pretrained("zaid646/multimodal-vision-agent-lora")


# --- Step 4: Define prediction function ---
def predict_action(instruction: str) -> dict:
    prompt = f"### Human: {instruction}\n### Assistant:"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=80,
            temperature=0.1,
            do_sample=True,
        )
    response = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:],
        skip_special_tokens=True,
    ).strip()
    print(f"Raw model output: {response}")
    match = re.search(r"<action>(.*?)</action>", response, re.DOTALL)
    if match:
        return json.loads(match.group(1))
    return {"action": "done"}


# --- Step 5: Test with various instructions ---
print(predict_action("Click the login button"))
# Expected: {'action': 'click', 'bbox': [450, 380, 120, 40]}

print(predict_action("Type email into the email field"))
# Expected: {'action': 'type', 'xpath': '//input[@name="email"]', 'text': '[email protected]'}

print(predict_action("Navigate to settings"))
# Expected: {'action': 'navigate', 'url': '/settings'}

print(predict_action("Scroll down the page"))
# Expected: {'action': 'scroll', 'direction': 'down'}

print(predict_action("Stop"))
# Expected: {'action': 'done'}

Full Agent Integration

For the complete agent loop with Playwright browser sandbox, LangGraph state machine, and evaluation harness, clone the GitHub repository:

git clone https://github.com/ZAID646/qwen2.5-vl-7b-playwright-desktop-lora.git
cd qwen2.5-vl-7b-playwright-desktop-lora

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install --upgrade pip
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium
playwright install-deps chromium

# Run all unit tests (no GPU required for MockVLM mode)
pytest -v

---

Real-World Test Results

The v2 adapter was tested against 4 real-world scenarios on actual websites using Playwright in headless Chromium mode on an RTX 4090. Each test captured before/after screenshots.

Test 1: GitHub Login

The model was instructed to fill the username and password fields on the GitHub login page (https://github.com/login).

Model output format: The v2 adapter produces semantic XPath selectors (//input[@name='username']) instead of brittle raw paths seen in v1 (/html/body/div/div/form/div[1]/input).

Test 2: HTTPBin Form

The model was instructed to fill name and email fields on https://httpbin.org/forms/post.

Test 3: Scroll

The model was instructed to scroll down on a long GitHub README page.

Test 4: Click Link

The model was instructed to click a link on https://example.com.

The model correctly identified the action type as click and attempted a CSS selector. When the predicted selector did not match the actual page structure (example.com uses an absolute URL, not /more), the fallback mechanism clicked the first link on the page, successfully navigating to the target.

---

v2 Improvements (vs v1)

Key Behavioral Changes

1. Semantic XPath Output: v1 produced rigid paths like /html/body/div/div/form/div[1]/input that break on any DOM change. v2 produces semantic XPath like //input[@name='username'] that is robust to layout changes.

2. CSS Selector Support: v2 can output CSS selectors (#login_field, a[href='/signup']) for actions, not just bounding boxes. This enables more precise element targeting.

3. Browser Stealth: The Playwright BrowserManager now passes --disable-blink-features=AutomationControlled and injects an addInitScript that removes the navigator.webdriver property. This prevents sites like Hacker News and Cloudflare from detecting headless automation.

4. ActionNode Robustness: The agent's ActionNode now handles all output formats: bbox as list [x, y, w, h] or object {x, y, width, height}, xpath string, CSS selector string, text/value field variations, and scroll_direction/direction field name variations.

---

Training Details

Dataset

The training dataset consists of 28 instruction-output pairs covering all 6 supported actions with diverse output formats:

Each example is formatted as a text prompt:

### Human: Click the login button
### Assistant: <action>{"action":"click","bbox":[450,380,120,40]}</action>

Quantization

The base model is loaded in 4-bit NormalFloat4 (NF4) precision using BitsAndBytesConfig:

BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

This reduces the base model memory footprint from approximately 14 GB (FP16) to approximately 4 GB (NF4), enabling training on consumer GPUs with 24 GB VRAM.

LoRA Configuration

Trainable parameters: 5,046,272 out of 7,620,662,784 total (0.0662%).

Training Results

Training was conducted on an NVIDIA GeForce RTX 4090 (25.3 GB VRAM) with CUDA, PyTorch 2.6.0, and Hugging Face Transformers.

Final training loss: 0.033 — the model learns to emit correct structured actions for the 28 training examples with high confidence.

Training throughput: 1.76 steps/second, 3.52 samples/second, 79.49 seconds total for 140 steps (28 examples x 10 epochs / 2 batch size).

---

Full Project Structure

The complete agent framework is available on GitHub at ZAID646/qwen2.5-vl-7b-playwright-desktop-lora.

qwen2.5-vl-7b-playwright-desktop-lora/
├── LICENSE                     # Apache 2.0
├── README.md                   # Full project documentation
├── CONTRIBUTING.md             # Contribution guidelines
├── pyproject.toml              # Project metadata and dependencies
├── requirements.txt            # Pip dependencies
├── setup.sh                    # Vast.ai environment setup
│
├── config/
│   ├── model.yaml              # Model selection, quantization, LoRA params
│   ├── sandbox.yaml            # Browser viewport, timeouts, concurrency
│   └── mock_scenarios.json     # Mock VLM scenario definitions
│
├── scripts/
│   ├── run_agent.py            # Single-task agent runner
│   ├── run_harness.py          # Full evaluation harness runner
│   └── train_lora.py           # QLoRA training script
│
├── src/
│   ├── agent/
│   │   ├── state.py            # AgentState, VisionOutput, StepRecord
│   │   ├── graph.py            # LangGraph state machine builder
│   │   ├── nodes.py            # PerceptionNode, ActionNode, RouterNode
│   │   └── prompts.py          # System prompt templates
│   │
│   ├── vision/
│   │   ├── model.py            # Model loader with quantization
│   │   ├── processor.py        # Screenshot preprocessing
│   │   ├── quant.py            # Quantization configuration
│   │   └── mock.py             # MockVLM for offline testing
│   │
│   ├── sandbox/
│   │   ├── browser.py          # Playwright BrowserManager singleton
│   │   ├── actions.py          # Atomic browser actions
│   │   └── recorder.py         # Screenshot + DOM capture
│   │
│   ├── memory/
│   │   ├── context.py          # ContextCompressor
│   │   └── history.py          # Step history summarizer
│   │
│   ├── harness/
│   │   ├── scenarios.py        # Benchmark scenario definitions
│   │   ├── runner.py           # Async scenario executor
│   │   └── metrics.py          # TCR, SER, TFI, SCRR computation
│   │
│   └── training/
│       ├── dataset.py          # UIExample dataclass
│       └── lora.py             # LoRA configuration builder
│
└── tests/
    ├── test_agent.py           # Agent graph and nodes tests
    ├── test_vision.py          # MockVLM and processor tests
    ├── test_harness.py         # Metrics computation tests
    └── test_memory.py          # Context compression tests

---

Dependencies

Core dependencies for loading and using this adapter:

Optional dependencies for the full agent framework:

---

Repository Contents

---

License

This adapter is released under the Apache License 2.0. See the LICENSE file for the full text.

The base model Qwen/Qwen2.5-7B-Instruct is governed by its own license (Qwen License).

---

Citation

If you use this adapter in your research or work, please cite:

@software{multimodal_vision_agent_lora,
  author = {Zaid},
  title = {Multimodal Vision Agent -- LoRA Adapter for Desktop UI Automation},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/zaid646/multimodal-vision-agent-lora}
}

---

<div align="center"> Built with Hugging Face Transformers, PEFT, bitsandbytes, LangGraph, and Playwright. </div>