Perseus-Doc-VL-0712

Perseus-Doc-vl-0712

The Perseus-Doc-vl-0712 model is a fine-tuned version of Qwen2.5-VL-7B-Instruct, optimized for Document Retrieval, Content Extraction, and Analysis Recognition. Built on top of the Qwen2.5-VL architecture, this model enhances document comprehension capabilities with focused training on 450K image pairs from a mixture of captioning datasets, including 230K from Corvus-OCR-Caption-Mix dataset and other document modular datasets from modular combination of opensource datasets best for doc OCR captioning, image reasoning, visual analysis, working on all category of images with variational dimensions.

Key Enhancements

• Context-Aware Multimodal Extraction and Linking for Documents: Advanced capability for understanding document context and establishing connections between multimodal elements within documents.

• Enhanced Document Retrieval: Designed to efficiently locate and extract relevant information from complex document structures and layouts.

• Superior Content Extraction: Optimized for precise extraction of structured and unstructured content from diverse document formats.

• Analysis Recognition: Specialized in recognizing and interpreting analytical content, charts, tables, and visual data representations.

• State-of-the-Art Performance Across Resolutions: Achieves competitive results on OCR and visual QA benchmarks such as DocVQA, MathVista, RealWorldQA, and MTVQA.

• Video Understanding up to 20+ minutes: Supports detailed comprehension of long-duration videos for content summarization, Q&A, and multi-modal reasoning.

• Visually-Grounded Device Interaction: Enables mobile/robotic device operation via visual inputs and text-based instructions using contextual understanding and decision-making logic.

Quick Start with Transformers

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/Perseus-Doc-vl-0712", torch_dtype="auto", device_map="auto"
)

processor = AutoProcessor.from_pretrained("prithivMLmods/Perseus-Doc-vl-0712")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

[!important] For open data analysis dataset, the document's content is phrased for training with the Gemini 2.5-Pro and other models.

[!important] Model type: Experimental.

Intended Use

This model is intended for:

• Context-aware multimodal extraction and linking for complex document structures.
• High-fidelity document retrieval and content extraction from various document formats.
• Analysis recognition of charts, graphs, tables, and visual data representations.
• Document-based question answering for educational and enterprise applications.
• Extraction and LaTeX formatting of mathematical expressions from printed or handwritten content.
• Retrieval and summarization from long documents, slides, and multi-modal inputs.
• Multilingual document analysis and structured content extraction for global use cases.
• Robotic or mobile automation with vision-guided contextual interaction.

Limitations

• May show degraded performance on extremely low-quality or occluded images.
• Not optimized for real-time applications on low-resource or edge devices due to computational demands.
• Variable accuracy on uncommon or low-resource languages/scripts.
• Long video processing may require substantial memory and is not optimized for streaming applications.
• Visual token settings affect performance; suboptimal configurations can impact results.
• In rare cases, outputs may contain hallucinated or contextually misaligned information.