VLAA-Thinker-Qwen2.5VL-3B

SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

<p align="center"> 🌐 <a href="https://ucsc-vlaa.github.io/VLAA-Thinking/" target="_blank">Project Page</a> • <img src="./assets/ar.svg" alt="Arxiv Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="./assets/VLAA-Thinker.pdf" target="_blank">Arxiv</a> • 💻 <a href="https://github.com/UCSC-VLAA/VLAA-Thinking" target="_blank">Code</a> </p> <p align="center"> <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/collections/UCSC-VLAA/vlaa-thinker-67eda033419273423d77249e" target="_blank">VLAA-Thinker Family</a> • 🤔 <a href="https://huggingface.co/datasets/UCSC-VLAA/VLAA-Thinking" target="_blank">VLAA-Thinking Dataset</a> <!-- • <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2VL-7B-Zero" target="_blank">VLAA-Thinker-Qwen2VL-7B-Zero</a> --> </p> <p align="center"> <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2.5VL-3B" target="_blank"><b>VLAA-Thinker-Qwen2.5-3B</b></a> • <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2.5VL-7B" target="_blank"><b>VLAA-Thinker-Qwen2.5-7B</b></a> </p> <!-- <p align="center"> <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2VL-2B" target="_blank">VLAA-Thinker-Qwen2VL-2B</a> • <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2VL-7B-Zero" target="_blank">VLAA-Thinker-Qwen2VL-7B-Zero</a> • <img src="./assets/hg.svg" alt="Hugging Face Logo" style="height: 1em; vertical-align: middle; margin-right: 0.3em;"> <a href="https://huggingface.co/UCSC-VLAA/VLAA-Thinker-Qwen2VL-7B" target="_blank">VLAA-Thinker-Qwen2VL-7B</a> </p> -->

Both VLAA-Thinker-Qwen2.5-3B and VLAA-Thinker-Qwen2.5-7B achieve SOTA performance on OpenCompass Multimodal Reasoning Leaderboard as of April 7th, 2025. <img src="assets/opencompass_4b_box.png" width = "640" alt="pipeline" align=center />

---

<img src="assets/opencompass_7b_box.png" width = "640" alt="pipeline" align=center />

Contents

• Quick Start 🚀
• Dataset Generation Pipeline 🏭
• Dataset Card 📚
• Examples 📝
• GRPO with Mixed Reward 💡
• Contributors 🙌

Quick Start 🚀

Inference

Run python inference.py. Note that our model is trained with a system prompt. Please ensure that it is included for inference.

Dataset Download

Run bash ./utils/download_dataset.sh. Specify the dataset root with absolute path. The dataset should be ordered as follows:

├── VLAA-Thinking-SFT-126K.json
├── VLAA-Thinking-GRPO-25K.json
└── images
    ├── allava_laion
    ├── arxivqa
    ├── chartqa
    ├── clevr_math
    ├── coco
    │   └── train2017
    ├── docvqa
    ├── geoqa170k
    ├── synthesis
    ├── vg
    │   ├── VG_100K
    │   └── VG_100K_2
    └── vizwiz

Training

Code coming soon!

VLAA-Thinking Data Generation 🏭

<img src="assets/data_generation.png" width = "640" alt="pipeline" align=center /> <details> <summary><strong>Step 1: Metadata Collection</strong></summary> We gather metadata from 9 distinct vision-language datasets, each comprising either closed- or open-ended visual questions. Unique images are sampled and processed through our comprehensive pipeline from datasets including CLEVR-Math, Math PUMA, ArxivQA, DocVQA, VizWiz, and ALLaVA. We directly utilize COCO and VisualGenome data as provided by LLaVA-CoT. Note that GeoQA170K is included exclusively in the RL dataset due to captioning hallucination challenges. Dataset statistics are summarized in the provided table. </details> <details> <summary><strong>Step 2: Visual Captioning and Additional Context</strong></summary> Each datapoint begins with an image-question-answer triplet. To effectively connect visual content with textual reasoning, we generate detailed, structured captions for each image using GPT-4o. Additionally, we leverage dataset-specific annotations to enrich the visual understanding: CLEVR-Math includes scene synthesis instructions, Math PUMA provides descriptive textual math problems, and ALLaVA-LAION offers carefully verified GPT-4V captions (prompt <a href="assets/prompts/1.captioning.txt" target="_blank">here</a>). </details> <details> <summary><strong>Step 3: Reasoning Answer Distillation</strong></summary> Utilizing the text-only reasoning model DeepSeek-R1, we generate structured and logical reasoning steps alongside final answers. The model reasons based on image captions, visual questions, and supplementary dataset-specific information, outputting a step-by-step rationale enclosed by explicit <think> tags for clarity (prompt <a href="assets/prompts/2.r1cot.txt" target="_blank">here</a>). </details> <details> <summary><strong>Step 4: Answer Refinement and Rewriting</strong></summary> To ensure clarity, consistency, and modality independence, we refine the reasoning outputs through a rewriting module powered by GPT-3.5-turbo. This refinement removes unnecessary modality-specific phrases and standardizes the answers. Samples exhibiting large textual deviations post-rewriting (more than 15 words difference) are filtered out to preserve minimal textual alteration (prompt <a href="assets/prompts/3.rewrite.txt" target="_blank">here</a>). </details> <details> <summary><strong>Step 5: Automated Verification</strong></summary> We verify the rewritten answers against the original ground-truth answers through an automated validation module. Only samples with verified correct answers are retained in the final training set, ensuring accuracy and logical coherence (prompt <a href="assets/prompts/4.verify.txt" target="_blank">here</a>). </details> <details> <summary><strong>Step 6: Curating Data Splits for SFT and RL</strong></summary> We partition the dataset into two mutually exclusive subsets tailored specifically for supervised fine-tuning (SFT) and reinforcement learning (RL). Following insights from recent studies highlighting RL's strength in deeper reasoning challenges, we use explicit self-reflective cues (termed "aha moments") in reasoning as a proxy for difficulty. Simpler examples without these cues populate the SFT set, while more challenging examples containing reflective "aha moments" form the dedicated RL dataset. </details> <!-- ### Captioning To enable the text-only R1 model to understand visual content, we need a detailed caption for each image. In our experiments: 1) For datasets with captions included (ALLaVA), we use the original captions; 2) For datasets containing *textual questions* that are equivalent to *image + visual questions* (Synthesis), we skip this step. 3) For the remaining datasets (CLEVR-Math, ArxivQA, GeoQA170K) which only contain images and VQAs, we generate a detailed caption for each image with **GPT-4o** (prompt [here](assets/prompts/1.captioning.txt)). --> <!-- ### Visual-Language CoT Generation In this step, we generate the Vision-Language CoT data from **R1**. 1) For datasets with textual questions (Synthesis), we prompt R1 with only the textual questions without any additional instructions/information. 2) For the other datasets, we prompt R1 with a caption and its associated visual question. We also intruct R1 not to output terms like "based on the caption" in its response because there's no caption presented during visual instruction tuning (prompt [here](assets/prompts/2.r1cot.txt)). --> <!-- ### Answer Rewriting After obtaining answers with CoT from R1, we observe that some answers contain undesired expressions, despite R1 instructed not to do so. Therefore, we use **GPT-4o** as a content rewriter to fix the errors while keeping everything else unchanged (prompt [here](assets/prompts/3.rewrite.txt)). --> <!-- ### Answer Verification The last step is to verify the correctness of generated responses. We leverage **GPT-3.5-Turbo** to compare the generated content with groundtruth from the original dataset (prompt [here](assets/prompts/4.verify.txt)). -->

Dataset Card 📚

Data statistics of VLAA-Thinking. We present the original volume of metadata (<span>#Original</span>), the data size after the distillation pipeline (<span>#Pipeline</span>), the size of sampled examples for SFT (<span>#Final SFT</span>) and RL (<span>#Final RL</span>), respectively. Note that we only use GeoQA170K with verifiable answers for the RL split.

Examples 📝

<details><summary>CLEVR-Math</summary> <img src="assets/examples/clevr_math-CLEVR_train_027977.png" align=center />

{
    "id": "clevr_math-CLEVR_train_027977.png",
    "image": "clevr_math/CLEVR_train_027977.png",
    "question": "Subtract all balls. How many objects are left?",
    "gt": 3,
    "ds_answer": "<think>  
Okay, let's see. The user is asking to subtract all balls and determine how many objects are left. First, I need to figure out which objects are considered \"balls\" in the image. The image shows a large blue metal sphere. A sphere is a three-dimensional circle, which is essentially a ball shape. So the blue metal sphere is definitely a ball.  

Next, the surrounding objects are two small red metal cylinders and a single small brown metal cylinder in the foreground. Cylinders are not balls; they have a different shape—circular bases and straight sides. So those would not be subtracted. The question says to subtract all balls, so only the sphere (ball) needs to be removed.  

Originally, there's the sphere (1 object), two red cylinders (2 objects), and one brown cylinder (1 object). Total objects before subtraction: 1 + 2 + 1 = 4. Subtracting the one ball leaves 3 objects: the two red cylinders and the brown cylinder.  

Wait, but I should make sure there are no other balls mentioned. The image only refers to the sphere as a ball-like object. The cylinders are clearly not balls. So subtracting the sphere leaves three cylinders. The answer should be 3.  
</think>  

In the scene, there is one large blue sphere (a ball-shaped object) centrally positioned. The remaining objects consist of two small red metal cylinders flanking it symmetrically and a single brown cylinder in the foreground. Removing the sphere leaves only the three cylindrical objects visible.  

**Answer:** There are 3 objects remaining.  ",
    "caption": "The image depicts an indoor setting with a smooth, neutral-colored surface background. Positioned centrally is a large blue metal sphere with a glossy finish, reflecting slight highlights. Surrounding it are two small red metal cylinders placed symmetrically on either side, also showing a metallic sheen. In the foreground, a single small brown metal cylinder is situated, completing the arrangement. The metals' shiny surfaces create a play of subtle reflections and shadows on the even surface beneath.",    "meta": {
        "caption_model_id": "gpt-4o-0806-nofilter-global",
        "rationale": "There are a large blue metal sphere, a small red metal cylinder, a small red metal cylinder, a small brown metal cylinder in the image."
    }
}

</details> <details><summary>GeoQA</summary> <img src="assets/examples/geoqa170k-geoqa_plus-2905.png" align=center />

{
    "id": "geoqa170k-geoqa_plus-2905-63590",
    "image": "geoqa170k/geoqa_plus-2905.png",
    "question": "Given a semicircular figure with the diameter represented by AB and intersecting chords AD and BC at point P, where angle DPB measures 60.0 degrees, calculate the value of tan(angle ADC) under the condition that D denotes the midpoint of the arc BC.
Choices:
A: \frac{1}{2}
B: 2
C: \frac{√{3}}{3}
D: √{3}",
    "gt": "C",
    "ds_answer": "<think>
Okay, so I'm trying to solve this geometry problem. Let me see if I can visualize it correctly. There's a semicircle with diameter AB. Points C and D are on the curved edge, and chords AC and BD intersect at point P. Below AB, there's the center O. The angle DPB is 60 degrees, and D is the midpoint of arc BC. I need to find the value of tan(angle ADC).

First, I should probably draw this out to get a better sense. I imagine a semicircle with AB as the diameter. O is the center, so AO = OB. Points C and D are on the semicircle, with D being the midpoint of arc BC. That means arc BD is equal to arc DC. Since it's a semicircle, the total arc from B to C is 180 degrees, so each arc BD and DC would be 90 degrees each. Wait, no, because D is the midpoint of arc BC, which is part of the semicircle. So if BC is some arc, D is halfway along it. But I'm not sure about the exact measure yet.

Chords AC and BD intersect at P, and angle DPB is 60 degrees. I remember that when two chords intersect, the products of the segments are equal. So AP * PC = BP * PD. But I'm not sure if that's directly useful here.

Since D is the midpoint of arc BC, it might have some special properties. In a semicircle, the midpoint of an arc would be equidistant from B and C. Also, angles subtended by the same arc are equal. Maybe I can use some properties of cyclic quadrilaterals here, but since it's a semicircle, all points lie on the circumference, so ABCD is a cyclic quadrilateral.

Wait, but ABCD is a quadrilateral inscribed in a semicircle, which makes it a cyclic quadrilateral. In such cases, opposite angles sum to 180 degrees. But I'm not sure if that's applicable here because we're dealing with a semicircle, not a full circle.

Let me think about the angles. Angle DPB is 60 degrees. Since P is the intersection of chords AC and BD, maybe I can use the property that the angle between two chords is equal to half the sum of the measures of the intercepted arcs. So angle DPB would be half the sum of arcs DB and AC. But I'm not sure about the exact arcs involved.

Alternatively, maybe I can use triangle properties. Since D is the midpoint of arc BC, triangle DBC might be isosceles. Wait, no, because D is on the semicircle, so triangle DBC is actually a right triangle because any triangle inscribed in a semicircle is a right triangle. So angle BDC is 90 degrees.

Wait, that's an important point. Since D is on the semicircle, triangle BDC is right-angled at D. So angle BDC is 90 degrees. That might help in finding angle ADC.

I need to find tan(angle ADC). Let's denote angle ADC as θ. So tan(θ) = opposite/adjacent. To find this, I might need to find the lengths of the sides of triangle ADC or use some trigonometric identities.

Let me consider the coordinates approach. Let's place the semicircle on a coordinate system with center O at (0,0), and AB as the diameter along the x-axis from (-r,0) to (r,0), where r is the radius. Then point D, being the midpoint of arc BC, would be somewhere on the semicircle. But without knowing the exact position of C, this might be tricky.

Alternatively, maybe I can use power of a point. Since P is the intersection of chords AC and BD, the power of point P with respect to the circle is equal for both chords. So PA * PC = PB * PD. But without specific lengths, I'm not sure how to proceed.

Wait, maybe I can assign some variables. Let's let the radius of the semicircle be r. Then AB = 2r. Let me assume r = 1 for simplicity, so AB = 2. Then O is at (0,0), A is at (-1,0), B is at (1,0). Let me place point C somewhere on the semicircle, say at an angle θ from the x-axis. Then point D, being the midpoint of arc BC, would be at an angle (θ + 180)/2 from the x-axis, but I'm not sure.

Alternatively, since D is the midpoint of arc BC, arc BD equals arc DC. So if I consider the central angles, angle BOD equals angle DOC. Since the total arc BC is some measure, say 2α, then each arc BD and DC would be α. But I'm not sure about the exact measure.

Wait, maybe I can use the fact that angle DPB is 60 degrees. Using the intersecting chords theorem, angle DPB is equal to half the sum of the measures of the intercepted arcs. So angle DPB = 1/2 (arc DB + arc AC) = 60 degrees. Therefore, arc DB + arc AC = 120 degrees.

But I'm not sure about the measure of arc AC. Alternatively, maybe I can express arc AC in terms of other arcs. Since the total semicircle is 180 degrees, arc AB is 180 degrees. Arc AC is part of that, so arc AC = 180 - arc CB. But I'm not sure.

Alternatively, since D is the midpoint of arc BC, arc BD = arc DC. Let's denote arc BD = arc DC = x. Then arc BC = 2x. Since the total semicircle is 180 degrees, arc AB = 180 degrees, so arc AC = 180 - arc BC = 180 - 2x.

Now, from the intersecting chords theorem, angle DPB = 1/2 (arc DB + arc AC) = 60 degrees. So 1/2 (x + (180 - 2x)) = 60. Simplifying, 1/2 (180 - x) = 60. Multiply both sides by 2: 180 - x = 120. Therefore, x = 60 degrees.

So arc BD = arc DC = 60 degrees, and arc BC = 120 degrees. Then arc AC = 180 - 120 = 60 degrees.

Now, I need to find tan(angle ADC). Let's consider triangle ADC. Since D is on the semicircle, angle ADC is an inscribed angle. The measure of angle ADC is half the measure of arc AC. Since arc AC is 60 degrees, angle ADC = 30 degrees.

Therefore, tan(angle ADC) = tan(30 degrees) = 1/√3 = √3/3.

Wait, but let me double-check. If angle ADC is 30 degrees, then tan(30) is indeed √3/3. So the answer should be option C: √3/3.
</think>

The value of tan(angle ADC) is \(\frac{\sqrt{3}}{3}\).

**Step-by-Step Explanation:**

1. **Understanding the Problem:**
   - We have a semicircle with diameter AB and center O.
   - Points C and D are on the semicircle, with D being the midpoint of arc BC.
   - Chords AC and BD intersect at point P, forming an angle DPB of 60 degrees.
   - We need to find tan(angle ADC).

2. **Key Properties and Theorems:**
   - **Intersecting Chords Theorem:** The measure of an angle formed by two intersecting chords is half the sum of the measures of the intercepted arcs.
   - **Midpoint of Arc:** Since D is the midpoint of arc BC, arcs BD and DC are equal.
   - **Inscribed Angle Theorem:** An angle inscribed in a circle is half the measure of its intercepted arc.

3. **Applying the Intersecting Chords Theorem:**
   - Angle DPB is formed by chords BD and AC, so:
     \[
     \angle DPB = \frac{1}{2} (\text{arc } DB + \text{arc } AC)
     \]
   - Given \(\angle DPB = 60^\circ\), we have:
     \[
     60^\circ = \frac{1}{2} (\text{arc } DB + \text{arc } AC)
     \]
     \[
     \text{arc } DB + \text{arc } AC = 120^\circ
     \]

4. **Determining Arc Measures:**
   - Since D is the midpoint of arc BC, arcs BD and DC are equal. Let each be \(x\).
   - Thus, arc BC = \(2x\).
   - The total semicircle is \(180^\circ\), so:
     \[
     \text{arc } AC = 180^\circ - \text{arc } BC = 180^\circ - 2x
     \]
   - Substituting into the earlier equation:
     \[
     x + (180^\circ - 2x) = 120^\circ
     \]
     \[
     180^\circ - x = 120^\circ
     \]
     \[
     x = 60^\circ
     \]
   - Therefore, arcs BD and DC are each \(60^\circ\), and arc AC is \(60^\circ\).

5. **Finding Angle ADC:**
   - Angle ADC is an inscribed angle intercepting arc AC.
   - By the Inscribed Angle Theorem:
     \[
     \angle ADC = \frac{1}{2} \times \text{arc } AC = \frac{1}{2} \times 60^\circ = 30^\circ
     \]

6. **Calculating the Tangent:**
   - \(\tan(30^\circ) = \frac{\sqrt{3}}{3}\).

**Answer:** \(\boxed{\frac{\sqrt{3}}{3}}\) (Option C)",
    "caption": "The image depicts a geometric figure set against a plain background. The main components of this figure include a semicircle with its diameter labeled \"AB.\" Inside the semicircle, two additional points \"C\" and \"D\" are marked on the curved edge, creating a chord, with lines \"AC\" and \"BD\" intersecting at point \"P.\" Below the diameter \"AB,\" there is a point \"O\" which appears to be the center of the semicircle. The semicircle is drawn with bold, continuous lines. No colors or textures are visible, and there are no textual annotations or numerical labels other than the letters.",    "meta": {
        "caption_model_id": "gpt-4o-0806-nofilter-global",
        "rationale": ""
    }
}

</details> <details><summary>Synthesis</summary> <img src="assets/examples/synthesis-gen_func-func_sine_22135407_qtmb.png" align=center />

{
    "id": "synthesis-2",
    "image": "synthesis/gen_func-func_sine_22135407_qtmb.png",
    "question": "What is the function's equation?",
    "gt": "Step 1: Substitute the point (pi/10, 5) into the function:  5 = a * sin(-5 * (pi/10)).
Step 2: Simplify the equation: 5 = a * sin(-pi/2).
Step 3: Since sin(-pi/2) = -1, we have: 5 = a * -1.
Step 4: Solve for a: a = -5.
Step 5: Substitute the value of a back into the original function: y = -5 * sin(-5 * x).

†Answer: y = -5 * sin(-5 * x).",
    "ds_answer": "<think>
First, I recognize that the function given is \( y = a \cdot \sin(-5x) \), and it passes through the point \( \left( \frac{\pi}{10}, 5 \right) \).

To find the value of \( a \), I'll substitute the coordinates of the point into the equation.

Substituting \( x = \frac{\pi}{10} \) and \( y = 5 \), the equation becomes:
\[ 5 = a \cdot \sin\left(-5 \cdot \frac{\pi}{10}\right) \]

Simplifying the argument of the sine function:
\[ -5 \cdot \frac{\pi}{10} = -\frac{\pi}{2} \]

So, the equation now is:
\[ 5 = a \cdot \sin\left(-\frac{\pi}{2}\right) \]

I know that \( \sin\left(-\frac{\pi}{2}\right) = -1 \), so substituting this value in:
\[ 5 = a \cdot (-1) \]

Solving for \( a \):
\[ a = -5 \]

Finally, substituting \( a = -5 \) back into the original function gives:
\[ y = -5 \cdot \sin(-5x) \]
</think>

To determine the equation of the function \( y = a \cdot \sin(-5x) \) that passes through the point \( \left( \frac{\pi}{10}, 5 \right) \), follow these steps:

1. **Substitute the Point into the Equation:**

   Plug \( x = \frac{\pi}{10} \) and \( y = 5 \) into the equation:
   \[
   5 = a \cdot \sin\left(-5 \cdot \frac{\pi}{10}\right)
   \]

2. **Simplify the Argument of the Sine Function:**
   \[
   -5 \cdot \frac{\pi}{10} = -\frac{\pi}{2}
   \]
   So, the equation becomes:
   \[
   5 = a \cdot \sin\left(-\frac{\pi}{2}\right)
   \]

3. **Evaluate the Sine Function:**
   \[
   \sin\left(-\frac{\pi}{2}\right) = -1
   \]
   Substituting this back into the equation:
   \[
   5 = a \cdot (-1)
   \]

4. **Solve for \( a \):**
   \[
   a = -5
   \]

5. **Write the Final Equation:**
   Substitute \( a = -5 \) back into the original function:
   \[
   y = -5 \cdot \sin(-5x)
   \]

6. **Simplify the Negative Sign (Optional):**
   Since \( \sin(-\theta) = -\sin(\theta) \), the equation can also be written as:
   \[
   y = 5 \cdot \sin(5x)
   \]

**Final Answer:**
\[
\boxed{y = -5 \sin(-5x)}
\]",
    "caption": "",
    "meta": {
        "text_question": "The function y = a * sin(-5 * x) goes through the points (pi/10,5). What is the function's equation?"
    }
}

</details> <details><summary>ArxivQA</summary> <img src="assets/examples/arxivqa-1907.10497_1.jpg" align=center />

{
    "id": "arxivqa-cond-mat-23815",
    "image": "arxivqa/1907.10497_1.jpg",
    "question": "In figure (b), how does the repetition rate affect the ratio of 9th to