LLaVA-OneVision-2-8B-Instruct

LLaVA-OneVision-2-8B-Instruct

[!IMPORTANT] 🌟 The core work on this project is led by Glint Lab. For an interactive online experience, visit ve2s.ai.

A multimodal vision-language model that handles single images, multi-image, and video inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder.

The model is distributed as a HuggingFace transformers checkpoint with custom code (trust_remote_code=True).

Requirements

Base (image + frame-sampling video)

pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord

Optional: codec video backend

The model ships a second video backend (video_backend="codec") that replaces uniform frame sampling with codec-aware canvas packing driven by motion vectors and bit-cost — typically yielding stronger long-video accuracy at the same token budget. To enable it you need two extra pieces:

# 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction.
pip install codec-video-prep opencv-python

# 2. A working `ffmpeg` binary must be on PATH.
#    Verify with: ffmpeg -version

ffmpeg version: ffmpeg 4.4.x – 7.x is recommended.

The codec backend additionally needs POSIX flock (already present on Linux/macOS) for the on-disk result cache, and roughly 2 GB free disk under $ONLINE_CODEC_CACHE_DIR (defaults to $HF_HOME/online_codec) per processed video.

Quick start

The repository ships a ready-to-run demo_inference.py that covers both image and video paths.

# Image (default sample image; no auth required)
python demo_inference.py

# Image, custom file + prompt
python demo_inference.py --mode image --media /path/to/cat.jpg \
    --prompt "What is the cat doing?"

# Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory)
python demo_inference.py --mode video --media /path/to/clip.mp4 \
    --num-frames 16 --max-pixels 200704 \
    --prompt "Describe what happens in this video."

Programmatic use

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda",
).eval()

# ----- Image -----
image = Image.open("cat.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image in detail."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

# ----- Video -----
# Lower max_pixels if you hit OOM on long videos.
processor.video_processor.max_pixels = 200704

messages = [{"role": "user", "content": [
    {"type": "video"},
    {"type": "text", "text": "Describe what happens in this video."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(
    text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True,
    num_frames=16,  # exact frame count; or use target_fps / max_frames
)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Video — codec backend (recommended for long videos)

The codec backend is exposed as a single processor kwarg (video_backend="codec"). Everything else — canvas extraction via cv-preinfer, on-disk caching, patch-position bookkeeping, chat-template rewriting — happens inside processor(...):

# Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH.
messages = [{"role": "user", "content": [
    {"type": "video"},
    {"type": "text", "text": "Describe what happens in this long video."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = processor(
    text=[text],
    videos=["long_clip.mp4"],
    video_backend="codec",
    max_pixels=150000,          # per-canvas pixel budget; lower if OOM
    return_tensors="pt",
    padding=True,
    # Optional: override codec defaults from preprocessor_config.json
    # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4},
)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Defaults for the codec pipeline live in preprocessor_config.json under the "codec" key (target_canvas=32, group_size=32, images_per_group=4, patch=14, min_group_frames=8, max_group_frames=64); they can be overridden per call via codec_config={...}.

Short-video behaviour: if the input video has fewer frames than target_canvas requires (or fewer than min_group_frames), a UserWarning is emitted and inference proceeds with however many canvases cv-preinfer can actually form. For very short clips, falling back to the frame-sampling backend is usually a better choice.

Notes

• The vision tower is a OneVision-style encoder; the language backbone is Qwen3-8B.
• chat_template.jinja follows the Qwen3 chat format and emits <|vision_start|>...<|vision_end|> placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend).
• Two video backends are available via processor(..., video_backend=...): "frames" (default, uniform sampling) and "codec" (canvas packing via cv-preinfer, requires codec-video-prep + ffmpeg).
• Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends.

Citation

If you find LLaVA-OneVision-2 useful in your research, please cite the following paper:

@misc{an2026llavaonevision2nextgenerationperceptualintelligence,
  title         = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
  author        = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
  year          = {2026},
  eprint        = {2605.25979},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2605.25979}
}

@misc{an2025llavaonevision15fullyopenframework,
  title         = {LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author        = {Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Didi Zhu and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
  year          = {2025},
  eprint        = {2509.23661},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2509.23661}
}

@misc{tang2026onevisionencodercodecalignedsparsityfoundational,
  title         = {OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence},
  author        = {Feilong Tang and Xiang An and Yunyao Yan and Yin Xie and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Chunyuan Li and Shikun Feng and Changrui Chen and Huajie Tan and Ming Hu and Manyuan Zhang and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
  year          = {2026},
  eprint        = {2602.08683},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2602.08683}
}

License

Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see Qwen/Qwen3-8B.