granite-docling-258M-GGUF

Granite-Docling-258M

Granite-Docling-258M is a compact vision-language document model developed by IBM, designed for OCR, document conversion, layout understanding, and structured content extraction workflows. This repository contains GGUF quantized variants of the model optimized for efficient local inference using llama.cpp.

The model specializes in transforming complex documents into machine-readable representations while preserving semantic structure, document hierarchy, tables, forms, equations, code blocks, and layout relationships. The quantized formats significantly reduce memory requirements while maintaining strong document-processing capabilities, enabling deployment on consumer hardware, edge devices, and enterprise document-processing pipelines.

---

Model Overview

• Model Name: Granite-Docling-258M
• Base Model: ibm-granite/granite-docling-258M
• Architecture: Multimodal Document Understanding Model
• Parameter Count: 258 Million
• Modalities: Text, Image
• Primary Languages: English
• Developer: IBM
• License: Apache 2.0

---

Quantization Formats

This repository provides various GGUF quantized versions of the Granite-Docling-258M model optimized for efficient local inference using llama.cpp. Below are the details of the available quantization formats.

IQ3_M

• Size reduction of approx 62.97% (117 MB) compared to 16-bit (316 MB)
• Aggressive 3-bit quantization optimized for maximum memory efficiency
• Suitable for lightweight OCR pipelines and low-memory deployment environments
• Enables practical execution of document-understanding workloads on consumer hardware
• Complex layouts, dense tables, mathematical content, and highly structured extraction tasks may experience reduced fidelity compared to higher-precision variants

IQ4_NL

• Size reduction of approx 62.34% (119 MB) compared to 16-bit (316 MB)
• Advanced 4-bit non-linear quantization designed to better preserve OCR quality and document understanding performance
• Better suited for structured document conversion, layout-aware analysis, and information extraction workflows
• Designed to reduce quantization loss compared to more aggressive formats
• May require slightly increased computational overhead during inference

IQ4_XS

• Size reduction of approx 62.66% (118 MB) compared to 16-bit (316 MB)
• Balanced 4-bit quantization focused on efficient inference and dependable document-processing performance
• Provides a practical balance between memory efficiency, extraction quality, and runtime speed
• Suitable for PDF conversion, OCR workflows, document digitization, and structured content extraction
• Maintains stable performance across most practical document-understanding tasks

Q6_K

• Size reduction of approx 48.42% (163 MB) compared to 16-bit (316 MB)
• Higher-precision 6-bit K-Quant format designed to preserve document structure recognition and extraction quality
• Better suited for demanding workloads involving complex layouts, tables, equations, technical documents, and dense visual content
• Provides stronger output consistency and improved retention of the original model capabilities compared to lower-bit formats
• Requires higher memory resources but offers enhanced document-conversion fidelity and structured extraction quality

---

Training Background (Original Model)

Granite-Docling-258M is trained with an emphasis on document intelligence, OCR, layout understanding, and structured document conversion across diverse document formats.

Pretraining

• Large-scale multimodal pretraining across document-centric image and text datasets
• Focus on text recognition, layout understanding, and document representation learning
• Optimized for downstream OCR, document conversion, and document-analysis workloads

Document Understanding Optimization

• Optimized for OCR, document parsing, and structured content extraction
• Enhanced for preserving document hierarchy, layout relationships, and semantic structure
• Improved performance on tables, forms, equations, code blocks, and technical documents
• Designed to generate structured document representations suitable for downstream processing workflows

---

Key Capabilities

• Optical Character Recognition (OCR) Extracts textual information from scanned documents, images, and visual content.

• Document Understanding Interprets document structure, context, and relationships between textual elements.

• Layout Analysis Understands page layouts, sections, tables, forms, and structured document formats.

• Structured Information Extraction Supports extraction of key-value pairs, fields, metadata, and document content.

• Document Conversion Converts complex documents into structured machine-readable representations.

• Efficient Local Deployment Quantized variants enable practical document-processing inference on consumer hardware.

---

Usage Example

Using llama.cpp

./llama-mtmd-cli \
  -m SandlogicTechnologies/Granite-Docling-258M_IQ4_NL.gguf \
  --mmproj SandlogicTechnologies/mmproj-granite-docling-258M-f16.gguf \
  --image document.png \
  -p "Extract all tables and convert the document into structured markdown."

---

Recommended Usecases

• Document Digitization Convert scanned documents into searchable and structured formats.

• OCR Pipelines Extract text from images, scanned pages, and document archives.

• Structured Content Extraction Extract tables, forms, metadata, and key-value information from documents.

• Enterprise Document Processing Automate large-scale document conversion and information retrieval workflows.

• RAG Data Preparation Transform enterprise documents into structured formats suitable for retrieval systems.

• Research and Experimentation Evaluate OCR, layout analysis, and document-understanding workflows.

---

Acknowledgments

These quantized models are based on the original work by the IBM Granite development team.

Special thanks to:

• The IBM team for developing and releasing the Granite-Docling-258M model.
• Georgi Gerganov and the llama.cpp open-source community for enabling efficient quantization and inference via the GGUF format.

---

Contact

For questions, feedback, or support, please reach out at [email protected] or visit https://www.sandlogic.com/