reverse-instruct-1.3m

REVERSE Visual Instruct 1.3M

Dataset Summary

Dataset Type:
REVERSE Visual Instruct 1.3M is a GPT-generated instruction-following dataset designed for training hallucination-aware vision-language models (VLMs). It builds on the LLaVA Instruct 665K dataset and includes structured annotations to indicate model confidence. We introduce three special tokens:

• <SPAN>: marks the beginning of a key phrase
• </CN>: denotes a confident (grounded) phrase
• </UN>: denotes an unconfident (potentially hallucinated) phrase

Roughly 50% of the examples contain correct (grounded) phrases from the original LLaVA dataset, while the remaining 50% include hallucinated or incorrect phrases generated via GPT-4o-mini-0718 and rule-based augmentations.

Collection Date:
February 2025

Data Generation:
Incorrect examples were synthesized via a combination of GPT-4o-mini prompts and deterministic, rule-based edits. Correct examples are drawn directly from the original LLaVA Visual Instruct dataset. Please refer to our GitHub Repo for more details.

License:

• Augmented portion: MIT License
• Base dataset (LLaVA): CC BY 4.0
• Note: As both datasets use OpenAI APIs in the data generation process, users should comply with OpenAI's Terms of Use.

Project Page:
https://reverse-vlm.github.io

Support & Issues:
GitHub Issues

Intended Use

Primary Use Cases:

• Research on hallucination detection and mitigation in VLMs
• Development and benchmarking of trustworthy vision-language assistants
• Instruction tuning for multi-modal dialogue agents

Target Users:
Researchers, practitioners, and hobbyists working in computer vision, natural language processing, and multi-modal AI.