AITQE

<style> .inline-img { display: inline-block; /* 或者使用 display: inline-block; 以便能设置宽度和高度 */ } </style> <h2> <a href="https://github.com/hanhuang22/AITQE"> <img class="inline-img" src="https://cdn-uploads.huggingface.co/production/uploads/65d86142a3c18e931641be25/ZT5e7XI0tWBfny-YKfnSV.png" alt="Logo" width=40> Beyond Filtering:<br>Adaptive Image-Text Quality Enhancement for MLLM Pretraining </a> </h2>

arxiv: https://arxiv.org/abs/2410.16166

github: https://github.com/hanhuang22/AITQE

[2024.10.12] Release the inference code and pre-trained model of AITQE.

We propose the Adaptive Image-Text Quality Enhancer, AITQE, a model that dynamically assesses and enhances the quality of image-text pairs. The conventional method (a) discards low-quality samples in raw data, reducing the amount of pretraining data, while our AITQE (b) enhances low-quality samples, retaining the same volume of data for MLLMs pretraining.

<img src="https://cdn-uploads.huggingface.co/production/uploads/65d86142a3c18e931641be25/CvTD-H7fZSx8F1BZ3a-WY.png" alt="illus" width="800">

Specifically, for pairs exhibiting low quality-such as low semantic similarity between modalities or subpar linguistic quality, AITQE performs text rewriting, generating high-quality text based on the input image and the raw low-quality text.

Use the code from github:

python inference.py \
       --model_path /path/to/AITQE \
       --output_all
       --gpu_id 0 \
       --image_path ./figs/test.png \
       --caption "Some random text to the image like this is a test"

and get the following output:

<pre style="white-space: pre-wrap; word-wrap: break-word;"> {"Recaption": "A man stands in front of a checklist of customer service questions, including 'Do you take each customer seriously?' and 'Do you qualify customers properly?'", "Overall Score": "2<Overall>", "Overall Explanation": "The caption is vague and does not accurately describe the image or its content. It lacks detail and relevance to the checklist shown in the image.", "Text Quality Score": 3, "Text Quality Explanation": "The caption is grammatically correct but lacks clarity and relevance to the image. It is vague and does not provide a meaningful description.", "Image-Text Matching Score": 2, "Image-Text Matching Explanation": "The caption does not accurately describe the image, which features a checklist of customer service questions. The caption is unrelated to the content of the image.", "Object Detail Score": 2, "Object Detail Explanation": "The caption does not provide any details about the objects in the image, such as the checklist or the person in the background.", "Semantic Understanding Score": 2, "Semantic Understanding Explanation": "The caption fails to convey any understanding of the image's context or purpose, which is about customer service evaluation.", "Text/Chart Description Score": 2, "Text/Chart Description Explanation": "The caption does not describe the text in the image, which is a checklist of customer service questions."} </pre>