MINGLE-1.0

MINGLE 1.0: Modeling INterpersonal Group-Level Engagement

This repository contains the fine-tuned vision-language model used in our AAAI submission "MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenes." The model builds upon Qwen2-VL-7B with LoRA (rank=64), and is specifically trained to classify social affiliation between pairs of individuals based on visual and depth cues.

Additional Resources

For data files (CSV) and example scripts, please visit the companion GitHub repository:
👉 https://github.com/ResearchDataset/AAAI2025-Dataset

🔍 Task: Social Group Region Detection

Given a pair of individuals in a street-view scene, the model determines whether they are socially interacting. This task requires reasoning over:

• Body orientation
• Facial expressions
• Gestures
• Relative positioning
• Depth distance

Unlike traditional object detection, this task targets semantically complex visual regions defined by interpersonal interaction—not discrete physical entities.

📦 Model Summary

• Base model: Qwen2-VL-7B
• Fine-tuning method: LoRA (rank=64)
• Training data: 61k annotated image pairs (Balanced, Y/N only)
• Input modality: RGB + Depth
• Answer format: 'Yes', 'No', or 'Not sure' (text-based classification)

🧪 Example Usage

The model is used with structured prompts combining bounding box and depth values.

Example User Prompt:

In the first image, two individuals are highlighted with bounding boxes (each given as [x1, y1, x2, y2]):

Box1: <|box_start|> 135,117,194,263 <|box_end|>
Box2: <|box_start|> 286,103,328,220 <|box_end|>

In the second image, which is the depth view of the same scene, the same two individuals are highlighted again:

Box1’: <|box_start|> 135,117,194,263 <|box_end|>
Box2’: <|box_start|> 286,103,328,220  <|box_end|>

Note: Box1 and Box1’ represent the same person, and Box2 and Box2’ represent the same person.

Depth Values (from 0-255, where 0 means far and 255 means close; these values are critical for determining proximity): Box1 and Box1’ = 179, Box2 and Box2’ = 142, so the Depth Difference is 37

Carefully analyze both images by considering all visual cues. In particular, pay attention to the following cues:
	•	Body orientation
	•	Facial expressions
	•	Gestures
	•	Depth distance
	•	Relative positioning

Based on these details, determine whether the individuals are actively interacting (e.g., engaged in conversation or displaying clear interactive behavior) or if they are merely near each other without meaningful interaction.

Your output must contain exactly one choice: ‘Yes’, ‘No’, or ‘Not sure’ with no additional commentary.

📈 Performance

On a manually validated test set:

• F1 score: 0.72
• Precision (Yes): 0.78
• Recall (Yes): 0.79

See full details in the accompanying paper.

📁 Files

This repository includes:

• LoRA-adapted checkpoint files
• Configuration files (config.json, generation_config.json, etc.)
• Tokenizer files
• Example prompt templates

📝 Citation

To cite the accompanying paper, please refer to the official publication link once available.

⚙️ Fine-tuning Framework

This model was fine-tuned using the open-source framework lmms-finetune.

⚠️ Note

This model is released for research purposes only. It is not intended for surveillance or any harmful use case. The annotations and model outputs may reflect biases present in the data sources.