DeepCaption-VLA-V2.0-7B

DeepCaption-VLA-V2.0-7B

DeepCaption-VLA-V2.0-7B is an advanced fine-tuned version of Qwen2.5-VL-7B-Instruct, specialized for Image Captioning and Vision Language Attribution (VLA). This enhanced release focuses on generating precise, attribute-rich captions that capture visual properties, object attributes, and scene details across diverse image types and aspect ratios.

Version V2.0 introduces significant improvements in multilingual inference, delivering higher captioning quality and attribution accuracy in languages including Chinese (Zh), Thai (Th), and others.

Key Highlights

1. Vision Language Attribution (VLA): Fine-tuned to attribute and define visual properties of objects, scenes, and environments with greater semantic precision.
2. Detailed Object Definitions: Generates attribute-rich captions, offering deeper visual understanding compared to generic captioning models.
3. High-Fidelity Descriptions: Excels at describing general, artistic, technical, abstract, and low-context images with enhanced descriptive detail.
4. Robust Across Aspect Ratios: Maintains caption accuracy across various formats — wide, tall, square, or irregular.
5. Variational Detail Control: Supports both concise summaries and fine-grained visual attributions depending on prompt structure.
6. Enhanced Multilingual Inference (New in V2.0): Optimized for generating accurate and descriptive captions in multiple languages, including English, Chinese (Zh), Thai (Th), and more.
7. Built on Qwen2.5-VL Architecture: Leverages the multimodal reasoning capabilities and instruction-following strengths of Qwen2.5-VL-7B.

model type: experimental

---

Sample Inferences [en, zh, thai] - <span style="color:red;">[DeepCaption-VLA-V2.0-7B]</span>

---

Comparison of Inference: Qwen2.5-VL-7B vs. <span style="color:red;">DeepCaption-VLA-V2.0-7B</span>

---

Example of a Recommended System Instruction

CAPTION_SYSTEM_PROMPT = """
You are an AI assistant that rigorously follows this response protocol:

1. For every input image, your primary task is to write a **precise caption**. The caption must capture the **essence of the image** in clear, concise, and contextually accurate language.

2. Along with the caption, provide a structured set of **attributes** that describe the visual elements. Attributes should include details such as objects, people, actions, colors, environment, mood, and other notable characteristics.

3. Always include a **class_name** field. This must represent the **core theme or main subject** of the image in a compact format.  
   - Use the syntax: `{class_name==write_the_core_theme}`  
   - Example: `{class_name==dog_playing}` or `{class_name==city_sunset}`  

4. Maintain the following strict format in your output:
   - **Caption:** <one-sentence description>  
   - **Attributes:** <comma-separated list of visual attributes>  
   - **{class_name==core_theme}**

5. Ensure captions are **precise, neutral, and descriptive**, avoiding unnecessary elaboration or subjective interpretation unless explicitly required.

6. Do not reference the rules or instructions in the output. Only return the formatted caption, attributes, and class_name.
"""

---

Quick Start with Transformers

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/DeepCaption-VLA-V2.0-7B", torch_dtype="auto", device_map="auto"
)

processor = AutoProcessor.from_pretrained("prithivMLmods/DeepCaption-VLA-V2.0-7B")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image with detailed attributes and properties."},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

---

Intended Use

• Generating attribute-rich image captions for research, dataset creation, and AI training.
• Vision-language attribution for object detection, scene understanding, and dataset annotation.
• Supporting creative, artistic, and technical applications requiring descriptive image understanding.
• Captioning across varied aspect ratios, non-standard datasets, and multilingual contexts.

Limitations

• May over-attribute or infer properties not explicitly visible in ambiguous or low-resolution images.
• Caption tone and level of detail may vary depending on prompt phrasing.
• Not intended for filtered captioning tasks; explicit or sensitive content may still appear.
• Performance may degrade slightly on highly synthetic or abstract visual domains.