CowCorpus-llama3-llava-next-8b

Model Card for CowCorpus/CowCorpus-llama3-llava-next-8b

<!-- Provide a quick summary of what the model is/does. -->

This model is a fine-tuned version of lmms-lab/llama3-llava-next-8b trained on the CowCorpus dataset.

This model is designed for the task of Human Intervention Prediction in collaborative web navigation. Unlike standard autonomous agents, this model predicts when a human user needs to take control from an AI agent. It utilizes multimodal inputs (screenshots, DOM trees, and action history) to distinguish between safe autonomous execution and moments requiring human error correction, preference alignment, or assistance.

It achieves state-of-the-art performance among open-weights models on the CowCorpus benchmark, achieving a Perfect Timing Score (PTS) of 0.201, outperforming closed-source models like GPT-4o.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: CowCorpus Team (Huq et al.)
• Model type: Multimodal Causal Language Model
• Base model: lmms-lab/llama3-llava-next-8b
• Language: English
• License: Llama 3 Community License Agreement
• Paper: Modeling Distinct Human Interaction in Web Agents
• Repository: GitHub: oaishi/CowCorpus

Input Data

The model is trained on a rich, multimodal state representation:

1. Visual Screenshot: The pixel-level view of the current webpage.
2. UI Structure (AX Tree): The accessibility tree (textual representation of DOM).
3. Past Trajectory: The history of actions taken by the agent/human so far.
4. Proposed Next Action: The action that the autonomous agent intends to take. The model evaluates if this intent is erroneous.

How to Get Started

For inference code, prompt templates, and setup instructions, please refer to our GitHub Repository.

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was trained on CowCorpus, ontaining 400 collaborative trajectories across:

• Dataset Size: ~4,200 total steps (2,748 Agent steps, 1,476 Human steps).
• Task Diversity: 200 Standardized Tasks (Mind2Web) and 200 Free-form User Tasks.
• Annotations: Steps are labeled with ground truth human intervention followup.

Training Configuration

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

• Hyperparameters:
  • Learning Rate: Linear decay from 1e-5 to ~2e-9
  • Epochs: 6
  • Global Steps: 120
  • Batch Size: 1
  • Precision: bfloat16

Evaluation

The model was evaluated on the CowCorpus test set. We report Step Accuracy, Intervention metrics (Precision, Recall, F1), and the Perfect Timing Score (PTS), which measures the temporal accuracy of intervention predictions.

Note: All models are evaluated in a zero-shot setting without reasoning.

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

If you use this model or dataset, please cite our work: Paper incoming