Apollo-VL-Edge-3B

Apollo-VL-Edge-3B

Intelligence isn't about scale. It's about precision.

---

<p align="center"> <img src="./banner.png" width="75%"> </p>

Overview

Apollo-VL-Edge-3B is a highly efficient Vision-Language Model (VLM) engineered by Pluto AI Labs. Built upon the foundation of Qwen/Qwen2.5-VL-3B-Instruct, Apollo-VL brings data-center-grade multimodal understanding and structured visual reasoning to consumer-grade hardware, including 8GB-class MacBooks and single NVIDIA T4 GPUs.

Rather than relying solely on parameter scale, Apollo-VL focuses on the quality and consistency of multimodal instruction data. The model was fine-tuned on 161,562 standardized multimodal instruction examples meticulously aggregated from high-quality open-source datasets.

Apollo-VL is optimized for visual tasks involving:

• 📊 Charts and Graphs
• 📐 Scientific Diagrams
• 🖥️ User Interfaces and Screenshots
• 📄 Dense Documents
• 🔎 OCR and Text Extraction
• 🧮 Visual Mathematics
• 🧠 Multi-step Visual Question Answering

While the base Qwen2.5-VL-3B-Instruct is a capable generalist, Apollo-VL is purpose-built for precision and structured visual analysis.

---

Key Distinctions

🧠 Elite Data Standardization

Apollo-VL is fine-tuned on the Apollo-VL-Massive-Dataset, a unified multimodal dataset created by combining and standardizing high-quality examples from LLaVA-NeXT and The Cauldron. This ensures robust performance across documents, charts, diagrams, OCR, and mathematical reasoning.

⚙️ Fault-Tolerant MLOps

Apollo-VL was trained using a custom 2× NVIDIA Tesla T4 DDP QLoRA pipeline. The training infrastructure incorporates a proprietary three-layer cross-session checkpoint recovery system, allowing training to seamlessly resume across interrupted compute sessions and bypass constrained cloud GPU runtimes (e.g., Kaggle's 12-hour limits).

💻 Local-First Architecture

Apollo-VL provides standalone FP16 weights and GGUF quantizations for local inference. The model is designed to make capable multimodal reasoning accessible without requiring hyperscale data-center infrastructure.

---

Live Demonstration

When presented with a complex visual reasoning task, Apollo-VL is designed to analyze the visual information and articulate its logic before producing a final answer.

Example Prompt

"Analyze this chart carefully. I need to know the exact percentage growth between Q2 and Q3. Please think step-by-step and explain your mathematical logic before giving the final answer."

Apollo-VL Output

To determine the percentage growth between Q2 and Q3, we need to follow these steps:

1. Identify the revenue values for Q2 and Q3 from the chart.

• Q2: $200k
• Q3: $350k

2. Calculate the difference in revenue between Q3 and Q2.

Difference = Q3 Revenue − Q2 Revenue

Difference = $350k − $200k = $150k

3. Calculate the growth rate.

Growth Rate = (Difference / Q2 Revenue) × 100%

Growth Rate = ($150k / $200k) × 100% = 75%

Final Answer: The company's revenue grew by 75% from Q2 to Q3.

---

<p align="center"> <img src="./benchmark.png" width="100%"> </p>

🏆 Benchmark Performance & Evaluation

Apollo-VL-Edge-3B is rigorously evaluated across standard multimodal benchmarks using the official lmms-eval evaluation harness.

Despite operating at 3.0B parameters (<6GB VRAM), Apollo-VL-Edge-3B achieves a Top 3 global ranking in the Sub-5B Vision-Language category, matching or outperforming significantly larger 4B+ models from major research labs.

📊 Global Sub-5B VLM Comparison

🔍 Detailed Benchmark Breakdown

1. Chart Understanding & Financial Intelligence (ChartQA)

• Overall Score: 78.60%
• Augmented Split (Structured Data Extraction): 94.16%
• Human Split (Complex Visual Interpretation): 63.04%
• Demonstrates exceptional visual grounding on high-density financial plots, multi-bar graphs, and unstructured legends.

2. Dense Document OCR & Text Parsing (OCRBench)

• Total Score: 786 / 1000
• Outperforms Microsoft Phi-3.5-Vision (+91 pts) and Google PaliGemma 2 (+136 pts) in complex character recognition, scene text reading, and structured table digitization.

3. Scientific & Diagrammatic Reasoning (AI2D)

• Accuracy: 77.98%
• Surpasses InternVL2-4B (76.20%) and Phi-3.5-Vision (75.40%), providing strong multi-step reasoning across educational diagrams and scientific figures.

---

⚡ Hardware Footprint & Quantization Guidance

Apollo-VL-Edge-3B is engineered specifically for deployment on consumer-grade hardware, Apple Silicon, and edge devices.

💡 Deployment Recommendation: For production OCR and critical document parsing, FP16 or GGUF Q8_0 is recommended to preserve fine visual patch features. For mobile and low-memory edge deployments, Q4_K_M delivers a 4× speedup with minimal reasoning degradation.

---

🔬 Reproducibility & Evaluation Setup

To reproduce our evaluation results using the standard lmms-eval suite:

# Clone evaluation framework
git clone --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git

cd lmms-eval && pip install -e .

# Run standard benchmark suite
python3 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=Pluto-AI-Labs/Apollo-VL-Edge-3B,dtype=float16 \
    --tasks mathvista_testmini_cot,chartqa,ai2d,ocrbench \
    --batch_size 1 \
    --log_samples \
    --output_path ./eval_logs/apollo_vl_edge_3b

---

Technical Specifications

Apollo-VL-Edge-3B uses the native Qwen2.5-VL architecture and applies Parameter-Efficient Fine-Tuning (PEFT) to the language-model component while keeping the vision encoder frozen.

Image Input
     │
     ▼
Qwen2.5-VL Vision Encoder
     │
     │ (Frozen)
     ▼
Vision-Language Projector
     │
     ▼
Qwen2.5-VL 3B Language Model
     │
     │ (QLoRA Fine-tuned)
     ▼
Structured Visual Analysis
     │
     ▼
Final Response

Training Configuration

LoRA Target Modules

The following transformer modules were targeted during LoRA fine-tuning:

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

---

Dataset & Data Pipeline

Apollo-VL was fine-tuned on the Apollo-VL-Massive-Dataset. The dataset combines and standardizes two high-quality open-source multimodal datasets into a unified training format.

All retained examples were converted into a common Apollo-VL formatting schema to enable consistent training across the combined dataset.

---

Usage

Transformers Inference

The FP16 Transformers checkpoint is recommended for the highest available reasoning quality.

import torch

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B",
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load processor
processor = AutoProcessor.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B"
)

# Synchronize chat template for local inference
if processor.chat_template is None:
    processor.chat_template = processor.tokenizer.chat_template

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/your/image.png"
            },
            {
                "type": "text",
                "text": "Analyze this image carefully. Think step-by-step before answering."
            },
        ],
    }
]

# Apply chat template
text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Process visual information
image_inputs, video_inputs = process_vision_info(messages)

# Prepare inputs
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate
with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512
    )

# Decode
output_text = processor.batch_decode(
    output_ids,
    skip_special_tokens=True
)[0]

print(output_text)

GGUF / llama.cpp

For memory-constrained local deployments, GGUF quantizations are provided in the repository.

llama-server \
  -m Apollo-VL-Edge-3B-Q4_K_M.gguf \
  --mmproj mmproj-Apollo-VL-Edge-3B-f16.gguf

For complex visual reasoning, FP16 or Q8_0 is recommended to preserve the model's chain-of-thought capabilities.

---

Intended Use

Apollo-VL-Edge-3B is intended for research, experimentation, and practical development involving:

• Vision-language models and multimodal reasoning
• Visual question answering and chart analysis
• OCR and document intelligence
• UI understanding and visual mathematics
• Local AI assistants and Edge AI deployment
• Model distillation and efficient multimodal inference

---

Limitations & Safety

Apollo-VL-Edge-3B is an open-source research model and may produce incorrect outputs. Known limitations include:

• Hallucinating information not present in the image
• Mathematical errors during multi-step calculations
• Potential quality degradation after aggressive quantization (e.g., Q4_K_M)
• Reasoning traces should not automatically be interpreted as proof of correctness

Do not use Apollo-VL for high-stakes autonomous decision-making (medical, legal, financial) without appropriate human oversight and independent verification.

---

Acknowledgements

Apollo-VL-Edge-3B builds upon the foundational work of the Qwen Team at Alibaba. We also thank the open-source communities behind Hugging Face, Transformers, PEFT, llama.cpp, lmms-eval, LLaVA-NeXT, and The Cauldron.

---

About Pluto AI Labs

Pluto AI Labs is an independent open-source AI research lab focused on efficient intelligence, multimodal reasoning, model distillation, and edge deployment. We explore how capable AI systems can be made smaller, faster, and more accessible without requiring hyperscale infrastructure.

• GitHub: https://github.com/Pluto-AI-Labs
• Hugging Face: https://huggingface.co/Pluto-AI-Labs

---

Citation

If you use Apollo-VL-Edge-3B in your research or projects, please cite:

@misc{apollo_vl_edge_3b,
  title        = {Apollo-VL-Edge-3B: Elite Visual Reasoning on Edge Hardware},
  author       = {Siddharth N.R. and Pluto AI Labs},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/Pluto-AI-Labs/Apollo-VL-Edge-3B}
}

---

<div align="center">

Apollo-VL-Edge-3B

Precision over scale. Reasoning at the edge.

Built by Pluto AI Labs

</div>