mage-vl-GGUF

Mage-VL

Mage-VL is a codec-native, proactive-streaming multimodal foundation model developed by Microsoft for image and video understanding. Its visual encoder, Mage-ViT, is trained from scratch and is designed to process visual information using codec-derived spatial and temporal importance rather than relying solely on uniformly sampled video frames.

The model combines Mage-ViT with a Qwen3-4B-Instruct-2507 causal language backbone. It provides a unified architecture for image understanding, frame-sampled video analysis, codec-native video understanding, long-video processing, and proactive streaming interactions.

Mage-VL is designed to reduce visual-token consumption by more than 75% through codec-aligned visual sparsity while preserving spatial and temporal context. The model authors report up to 3.5× wall-clock inference speedup over uniform frame sampling in the relevant video-processing setting.

---

Model Overview

• Model Name: Mage-VL
• Base Model: microsoft/Mage-VL
• Architecture: Mage-ViT Vision Encoder + Qwen3-4B-Instruct-2507 Causal Decoder
• Vision Encoder: Mage-ViT, trained from scratch
• Language Backbone: Qwen3-4B-Instruct-2507
• Modalities: Text, Image, Video
• Primary Language: Multilingual
• Developer: Microsoft
• License: Apache 2.0

---

Quantization Formats

This repository provides GGUF-converted versions of the Mage-VL model optimized for efficient local inference. The conversion metadata supplied for this repository contains an F16 backbone and three I-Matrix quantization variants.

IQ3_M

• Size reduction of approx 75.91% (1.98 GB) compared to 16-bit (8.22 GB)
• Aggressive 3-bit quantization designed to substantially reduce the memory footprint of the Mage-VL backbone
• Suitable for memory-constrained local deployment where minimizing model storage and runtime memory is a primary requirement
• Enables practical experimentation with the model's language backbone using significantly fewer resources than the F16 representation
• Lower numerical precision can have a greater effect on detailed reasoning and generation behavior than the higher-precision variants

IQ4_NL

• Size reduction of approx 70.44% (2.43 GB) compared to 16-bit (8.22 GB)
• 4-bit non-linear quantization designed to provide a higher-precision representation than the supplied IQ3_M variant
• Provides a practical balance between model footprint and preservation of the underlying model's generation behavior
• Suitable for local inference scenarios where additional precision is preferred over the smallest available quantized representation
• Requires more storage and memory than IQ3_M

IQ4_XS

• Size reduction of approx 71.78% (2.32 GB) compared to 16-bit (8.22 GB)
• Compact 4-bit quantization providing an intermediate deployment option between the 3-bit and higher-precision representations
• Designed to reduce memory requirements while maintaining useful model behavior for local inference
• Suitable for resource-conscious deployments requiring a balance between model size and numerical precision
• Uses a smaller footprint than IQ4_NL while retaining a 4-bit representation

Q6_K

• Size reduction of approx 58.88% (3.38 GB) compared to 16-bit (8.22 GB)
• Higher-precision 6-bit K-Quant representation designed to retain more parameter precision than the supplied IQ3_M and 4-bit variants
• Suitable when model fidelity is prioritized over achieving the minimum possible memory footprint
• Provides a larger representation than the lower-bit GGUF variants while remaining substantially smaller than the supplied F16 backbone
• The higher bit depth is expected to reduce quantization-related information loss compared with the lower-bit variants

---

Training Background (Original Model)

Mage-VL uses a unified multimodal training approach combining a visual encoder trained from scratch with a pretrained Qwen3 language backbone. The model is trained to support image understanding, video reasoning, long-context visual processing, and proactive streaming within a single checkpoint.

Visual Pretraining

• Mage-ViT is trained from scratch as a codec-native visual encoder
• Uses a shared 16×16 patch grid with 3D rotary positional encoding
• Uses codec-derived information to identify visually important regions
• Supports traditional H.264/HEVC codec information through motion vectors and residual energy
• Supports neural DCVC-RT codec information through learned rate maps
• Designed to reduce redundant visual-token processing while maintaining spatio-temporal information

Multimodal Training

• Multimodal alignment using large-scale image and video caption datasets
• Instruction tuning with image and short-video data
• Expansion to medium- and long-duration video understanding
• Codec-native long-context adaptation using rolling codec windows
• Proactive streaming alignment through a lightweight cognition gate

---

Key Capabilities

• Image Understanding Processes visual inputs together with textual instructions for image-based question answering and visual reasoning.

• Video Understanding Analyzes temporal visual information across short, long, and ultra-long video sequences.

• Codec-Native Video Processing Uses codec-derived motion and residual information to focus computation on visually important regions.

• Long-Context Video Analysis Processes extended video sequences through variable-length visual token streams while maintaining temporal relationships.

• Proactive Streaming Uses a lightweight cognition gate to monitor rolling visual windows and activate the full multimodal model when a response-worthy event is detected.

• Multimodal Reasoning Combines projected visual representations with textual tokens through a shared causal Qwen3 decoder.

• Efficient Visual Tokenization Codec-aligned visual sparsity reduces redundant visual tokens and is designed to improve video-processing efficiency.

• Unified Multimodal Model A single Mage-VL checkpoint supports image understanding, video understanding, codec-based processing, and proactive streaming rather than requiring separate model variants.

---

Usage Example

Using llama.cpp

./llama-cli \
  -m SandLogicTechnologies/Mage-VL-backbone-IQ4_NL.gguf \
  -p "Describe the important visual information in this input."

The supplied GGUF conversion files are named Mage-VL-backbone-*; multimodal image/video processing may require additional vision and video processing components supported by the selected inference backend. The original Mage-VL repository provides Transformers and vLLM usage paths for the complete multimodal model.

---

Recommended Usecases

• Image Understanding Analyze images and answer questions involving visual content.

• Video Understanding Perform temporal reasoning and question answering over video sequences.

• Long-Video Analysis Process extended video content while preserving temporal context.

• Codec-Aware Video Analytics Develop applications that exploit codec-derived motion and residual information for efficient visual processing.

• Streaming Perception Build systems that continuously monitor visual streams and respond when significant events occur.

• Multimodal AI Assistants Develop assistants capable of combining visual and textual information.

• Video Event Detection Identify response-worthy events within continuous video streams.

• Multimodal Research Experiment with codec-native visual representation, efficient video tokenization, long-context video understanding, and proactive multimodal interaction.

---

Acknowledgments

These quantized models are based on the original work by the *Microsoft- development team.

Special thanks to:

• The Microsoft team for developing and releasing the Mage-VL model and Mage-ViT codec-native visual architecture.

• The Qwen team for developing the Qwen3-4B-Instruct-2507 language backbone used by Mage-VL.

• Georgi Gerganov and the llama.cpp open-source community for enabling efficient quantization and inference via the GGUF format.

---

Contact

For questions, feedback, or support, please reach out at [email protected] or visit https://www.sandlogic.com/.