Ovis2.6-30B-A3B-AWQ-4bit

<div align="left"> <a href="https://cyan.kiwi" target="_blank"><img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-cyankiwi.png" width="800" alt="cyankiwi, the LLM inference agent for open models"></a> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Website</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://cyan.kiwi/" target="_blank">cyan.kiwi</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Models</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/cyankiwi" target="_blank">Hugging Face</a> · <a href="https://www.modelscope.ai/organization/cyankiwi" target="_blank">ModelScope</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Connect</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://x.com/cyankiwi_ai" target="_blank">X</a> · <a href="https://www.reddit.com/r/cyankiwi/" target="_blank">Reddit</a> · <a href="mailto:[email protected]">Email</a></td> </tr> </table> </div> <br> <div align="left"> <img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-awq.png" width="800" alt="cyankiwi Activation-aware Weight Quantization"> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Version</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://github.com/vllm-project/llm-compressor" target="_blank">llm-compressor</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Calibration</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset" target="_blank">nvidia/Llama-Nemotron-Post-Training-Dataset</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Model Size</b></td> <td align="left" style="border:none; padding:4px 0;">19.79 GB</td> </tr> </table> </div>

---

Ovis2.6-30B-A3B

<div align="center"> <img src=https://cdn-uploads.huggingface.co/production/uploads/637aebed7ce76c3b834cea37/3IK823BZ8w-mz_QfeYkDn.png width="30%"/> </div>

Introduction

We introduce Ovis2.6-30B-A3B, the latest advancement in the Ovis series of Multimodal Large Language Models (MLLMs). Building on the strong foundation of Ovis2.5, Ovis2.6 upgrades the LLM backbone to a Mixture-of-Experts (MoE) architecture, delivering superior multimodal performance at a fraction of the serving cost. It also brings major improvements in long-context and high-resolution understanding, visual reasoning with active image analysis, and information-dense document comprehension.

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/658a8a837959448ef5500ce5/IPsQk8gTTMD-ipTye3WED.png" width="100%" /> </div>

Key Features

• MoE Architecture: Superior Performance with Low Serving Cost
  The LLM backbone has been upgraded to a Mixture-of-Experts (MoE) architecture. This allows Ovis2.6 to scale up to 30B total parameters, capturing vast amounts of knowledge and nuance. Crucially, it achieves this with only ~3B active parameters during inference, ensuring low serving costs and high throughput.

• Enhanced Long-Sequence and High-Resolution Processing
  Ovis2.6 extends the context window to 64K tokens and supports image resolutions up to 2880×2880, significantly improving its ability to process high-resolution and information-dense visual inputs. These enhancements are particularly effective for long-document question answering, where the model must gather and synthesize clues scattered across multiple pages to derive the correct answer.

• Think with Image
  We introduce the "Think with Image" capability, which transforms vision from a passive input into an active cognitive workspace. During reasoning, the model can actively invoke visual tools (e.g., cropping and rotation) to re-examine and analyze image regions within its Chain-of-Thought, enabling multi-turn, self-reflective reasoning over visual inputs for higher accuracy on complex tasks.

• Reinforced OCR, Document, and Chart Capabilities
  Continuing our focus on information-dense visual tasks, we have further reinforced the model's capabilities in Optical Character Recognition (OCR), document understanding, and chart/diagram analysis. Ovis2.6 excels not only at accurately extracting structured information from visual data, but also at reasoning over the extracted content.

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/658a8a837959448ef5500ce5/3_A0CA-oO0Ie_WoigjAwo.png" width="100%" /> </div>

Performance

The following table presents a detailed performance comparison. Please note that superscripted results are sourced from external technical reports, and Qwen scores represent the highest value between its Think and Instruct versions. For quick reference, the best results are highlighted in red, and the second-best results are underlined. All values are rounded to one decimal place.

Quick Inference (vLLM)

uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
vllm serve AIDC-AI/Ovis2.6-30B-A3B --trust-remote-code --tensor-parallel-size 4
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "AIDC-AI/Ovis2.6-30B-A3B",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://cdn-uploads.huggingface.co/production/uploads/658a8a837959448ef5500ce5/TIlymOb86R6_Mez3bpmcB.png"}},
          {"type": "text", "text": "Calculate the sum of the numbers in the middle box in figure (c)."}
        ]
      }
    ]
  }'

Quick Inference (transformers)

Below is a simple example demonstrating how to run Ovis2.6 with a single image input.

First, install the required dependencies:

pip install torch==2.7.1 transformers==4.57.0 numpy==1.25.0 pillow==10.3.0 moviepy==1.0.3 accelerate==1.12.0
pip install --no-build-isolation --no-cache-dir flash-attn==2.8.3

Then, run the following code.

import torch
import requests
from PIL import Image
from transformers import AutoModelForCausalLM

# Thinking mode & budget
enable_thinking = True
enable_thinking_budget = True  # Only effective if enable_thinking is True.

# Total tokens for thinking + answer. Ensure: max_new_tokens > thinking_budget + 25
max_new_tokens = 2048
thinking_budget = 1024

model = AutoModelForCausalLM.from_pretrained(
    "AIDC-AI/Ovis2.6-30B-A3B",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": Image.open(requests.get("https://cdn-uploads.huggingface.co/production/uploads/658a8a837959448ef5500ce5/TIlymOb86R6_Mez3bpmcB.png", stream=True).raw)},
        {"type": "text", "text": "Calculate the sum of the numbers in the middle box in figure (c)."},
    ],
}]

input_ids, pixel_values, grid_thws = model.preprocess_inputs(
    messages=messages,
    add_generation_prompt=True,
    enable_thinking=enable_thinking
)
input_ids = input_ids.cuda()
pixel_values = pixel_values.cuda() if pixel_values is not None else None
grid_thws = grid_thws.cuda() if grid_thws is not None else None

outputs = model.generate(
    inputs=input_ids,
    pixel_values=pixel_values,
    grid_thws=grid_thws,
    enable_thinking=enable_thinking,
    enable_thinking_budget=enable_thinking_budget,
    max_new_tokens=max_new_tokens,
    thinking_budget=thinking_budget,
)

response = model.text_tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

The thinking and thinking budget logic can be applied in the same way for multi-image, video and pure text scenarios.

Note (answer extraction for CoT/Thinking): To make evaluation and usage easier, we recommend appending a fixed suffix to prompts when using chain-of-thought (CoT) or thinking mode. This ensures the model clearly outputs a final answer that can be extracted programmatically:

End your response with 'Final answer: '.

For example:

Calculate the sum of the numbers in the middle box in figure (c).
End your response with 'Final answer: '.

Tip: The sections below include an optional streaming helper (compatible with two-phase thinking/budget runs) and extra inference modes: multi-image, video, and text-only.

<details> <summary>Optional: Streaming (Advanced)</summary>

To support thinking budget, we modified the implementation of the Ovis generate method and the default TextIteratorStreamer is now incompatible. If you need to stream model output, be sure to use the helper class below.

# --- Budget-aware streamer helper ---
from transformers import TextIteratorStreamer

class BudgetAwareTextStreamer(TextIteratorStreamer):
    """A streamer compatible with Ovis two-phase generation.

    Call .manual_end() after generation to flush any remaining text.
    """
    def manual_end(self):
        if len(self.token_cache) > 0:
            text = self.tokenizer.decode(self.token_cache, **self.decode_kwargs)
            printable_text = text[self.print_len:]
            self.token_cache = []
            self.print_len = 0
        else:
            printable_text = ""
        self.next_tokens_are_prompt = True
        self.on_finalized_text(printable_text, stream_end=True)

    # Disable base class's end hook; we'll finalize via manual_end()
    def end(self):
        pass

Example usage:

streamer = BudgetAwareTextStreamer(
    model.text_tokenizer,
    skip_prompt=True,
    skip_special_tokens=True
)

outputs = model.generate(
    inputs=input_ids,
    pixel_values=pixel_values,
    grid_thws=grid_thws,
    enable_thinking=enable_thinking,
    enable_thinking_budget=enable_thinking_budget,
    max_new_tokens=max_new_tokens,
    thinking_budget=thinking_budget,
    streamer=streamer
)

</details> <details> <summary>Example: Multi-image</summary> Demonstrates how to run inference with multiple images and a related question.

# Multi-image inference
multi_image_files = [
    "/path/to/image_1.jpg",
    "/path/to/image_2.jpg",
    "/path/to/image_3.jpg",
]

content = [{"type": "image", "image": Image.open(p).convert("RGB")} for p in multi_image_files]
content.append({"type": "text", "text": "Describe the images."})
messages = [{"role": "user", "content": content}]

input_ids, pixel_values, grid_thws = model.preprocess_inputs(messages=messages, add_generation_prompt=True, max_pixels=896*896)
input_ids = input_ids.cuda()
pixel_values = pixel_values.cuda().to(model.dtype) if pixel_values is not None else None
grid_thws = grid_thws.cuda() if grid_thws is not None else None

with torch.no_grad():
    outputs = model.generate(inputs=input_ids, pixel_values=pixel_values, grid_thws=grid_thws,
                             max_new_tokens=1024, do_sample=True,
                             eos_token_id=model.text_tokenizer.eos_token_id,
                             pad_token_id=model.text_tokenizer.pad_token_id)
print(model.text_tokenizer.decode(outputs[0], skip_special_tokens=True))

</details> <details> <summary>Example: Video</summary> Demonstrates how to run inference on a video by sampling multiple frames and asking the model to describe the content.

# Video inference
from moviepy.editor import VideoFileClip  # pip install moviepy==1.0.3

video_file = "/path/to/video_1.mp4"
num_frames = 8

with VideoFileClip(video_file) as clip:
    total_frames = int(clip.fps * clip.duration)
    indices = [int(i * total_frames / num_frames) for i in range(num_frames)]
    frames = [Image.fromarray(clip.get_frame(t)) for t in (idx / clip.fps for idx in indices)]

messages = [{"role": "user", "content": [
    {"type": "video", "video": frames},
    {"type": "text", "text": "Describe this video in detail."},
]}]

input_ids, pixel_values, grid_thws = model.preprocess_inputs(messages=messages, add_generation_prompt=True, max_pixels=896*896)
input_ids = input_ids.cuda()
pixel_values = pixel_values.cuda().to(model.dtype) if pixel_values is not None else None
grid_thws = grid_thws.cuda() if grid_thws is not None else None

with torch.no_grad():
    outputs = model.generate(inputs=input_ids, pixel_values=pixel_values, grid_thws=grid_thws,
                             max_new_tokens=1024, do_sample=True,
                             eos_token_id=model.text_tokenizer.eos_token_id,
                             pad_token_id=model.text_tokenizer.pad_token_id)
print(model.text_tokenizer.decode(outputs[0], skip_special_tokens=True))

</details> <details> <summary>Example: Text-only</summary> Demonstrates how to run inference using only text input without any images or videos.

# Text-only inference
messages = [{"role": "user", "content": "Hi, please introduce Yellow Mountain."}]

input_ids, _, _ = model.preprocess_inputs(messages=messages, add_generation_prompt=True)
input_ids = input_ids.cuda()

with torch.no_grad():
    outputs = model.generate(inputs=input_ids, max_new_tokens=1024, do_sample=True,
                             eos_token_id=model.text_tokenizer.eos_token_id,
                             pad_token_id=model.text_tokenizer.pad_token_id)
print(model.text_tokenizer.decode(outputs[0], skip_special_tokens=True))

</details>

To enable grounding, end your prompt with Please provide the bounding box coordinates. (for boxes) or Please provide the point coordinates. (for points). To target a specific object, wrap its description in <ref> tags, e.g.:

Find the <ref>red apple</ref> in the image. Please provide the bounding box coordinates.

Coordinates are normalized to [0,1) with the origin (0,0) at the top-left corner of the image.

• Point: <point>(x,y)</point>
• Bounding box: <box>(x1,y1),(x2,y2)</box> where (x1,y1) is top-left, (x2,y2) is bottom-right.
• Multiple results can be listed in square brackets: [<box>(...)</box>,<box>(...)</box> ]

Example:

The image features a serene scene with <ref>three birds</ref>[
  <box>(0.401,0.526),(0.430,0.557)</box>,
  <box>(0.489,0.494),(0.516,0.526)</box>,
  <box>(0.296,0.529),(0.324,0.576)</box>
] flying in formation against a clear blue sky.

Citation

If you find Ovis useful, please consider citing the paper

@article{lu2025ovis25technicalreport,
  title={Ovis2.5 Technical Report}, 
  author={Shiyin Lu and Yang Li and Yu Xia and Yuwei Hu and Shanshan Zhao and Yanqing Ma and Zhichao Wei and Yinglun Li and Lunhao Duan and Jianshan Zhao and Yuxuan Han and Haijun Li and Wanying Chen and Junke Tang and Chengkun Hou and Zhixing Du and Tianli Zhou and Wenjie Zhang and Huping Ding and Jiahe Li and Wen Li and Gui Hu and Yiliang Gu and Siran Yang and Jiamang Wang and Hailong Sun and Yibo Wang and Hui Sun and Jinlong Huang and Yuping He and Shengze Shi and Weihong Zhang and Guodong Zheng and Junpeng Jiang and Sensen Gao and Yi-Feng Wu and Sijia Chen and Yuhui Chen and Qing-Guo Chen and Zhao Xu and Weihua Luo and Kaifu Zhang},
  year={2025},
  journal={arXiv:2508.11737}
}

@article{lu2024ovis,
  title={Ovis: Structural Embedding Alignment for Multimodal Large Language Model},
  author={Shiyin Lu and Yang Li and Qing-Guo Chen and Zhao Xu and Weihua Luo and Kaifu Zhang and Han-Jia Ye},
  year={2024},
  journal={arXiv:2405.20797}
}

License

This project is licensed under the Apache License, Version 2.0 (SPDX-License-Identifier: Apache-2.0).

Disclaimer

We used compliance-checking algorithms during the training process, to ensure the compliance of the trained model to the best of our ability. Due to the complexity of the data and the diversity of language model usage scenarios, we cannot guarantee that the model is completely free of copyright issues or improper content. If you believe anything infringes on your rights or generates improper content, please contact us, and we will promptly address the matter.