jina-ocr-v1-GGUF

jina-ocr-v1 GGUF

GGUF conversion of jinaai/jina-ocr-v1 for llama.cpp, so the model can run locally on CPU / Apple Silicon / any llama.cpp backend.

jina-ocr-v1 is an end-to-end document parsing model built on DeepSeek-OCR: a DeepEncoder vision tower (SAM-base + CLIP-L) that represents a 1024x1024 page in 256 visual tokens, plus a 3B mixture-of-experts decoder with ~570M active parameters per token. It scores 91.14 on OmniDocBench v1.6 and 83.4 on olmOCR-Bench.

[!IMPORTANT] These files need a one-line patch to llama.cpp to run. See Requirements. Without it you get GGML_ASSERT(fabs(freq_base - 10000.0) < 1e-4) failed.

Files

You always need the mmproj file plus exactly one decoder file.

[!WARNING] The jina-ocr-v1-Q8_0.gguf uploaded on 20 Sep 2026 was incomplete: its GGUF header was never written, so llama.cpp rejected it with invalid magic characters: '????', expected 'GGUF'. It was replaced on 12 Oct 2026. If you downloaded Q8_0 before then, download it again. The other three files were unaffected.

Requirements

jina-ocr-v1 sets rope_theta = 1000000 and a 32768-token context, while the DeepSeek-OCR checkpoint it derives from leaves rope_theta unset (HF defaults it to 10000). llama.cpp's DeepSeek-OCR graph still asserts the base model's value, even though the configured freq_base is passed to ggml_rope_ext on the very next line, so the assert has to go:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
sed -i.bak '/GGML_ASSERT(fabs(freq_base - 10000.0) < 1e-4);/d' src/models/deepseek2.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-mtmd-cli llama-server

Nothing else in llama.cpp needs changing to run these files.

DeepSeek-OCR itself is fully supported upstream (since #17400, March 2026). The assert is harmless for the reference deepseek-ai/DeepSeek-OCR checkpoint, which leaves rope_theta unset, and only trips for checkpoints that set it to 1e6, as this one does — which is why upstream still carries it.

Last verified against llama.cpp master 0f701c8 (12 Oct 2026): the assert is still present, and with the patch applied both llama-mtmd-cli and llama-server run these files unchanged.

Usage

--jinja is required: this checkpoint ships its own chat template, which is not one of llama.cpp's built-in legacy templates.

./build/bin/llama-mtmd-cli \
  -m jina-ocr-v1-Q8_0.gguf \
  --mmproj mmproj-jina-ocr-v1-F16.gguf \
  --jinja --temp 0 -ngl 99 \
  --image document.png \
  -p "Transcribe the provided document image into a clean Markdown format, preserving the natural reading order."

Or as an OpenAI-compatible server:

./build/bin/llama-server \
  -m jina-ocr-v1-Q8_0.gguf \
  --mmproj mmproj-jina-ocr-v1-F16.gguf \
  --jinja -ngl 99

Prompts

The default prompt above is what Jina recommends. The published benchmark numbers were measured with a stricter instruction (LaTeX math, HTML tables, drop headers/footers and figures) — see the original model card for its full text.

Limitations vs. the original

• No FastMTP. The upstream model ships a FastMTP speculative decoding head (K=3) that roughly doubles decode speed under vLLM. llama.cpp's MTP support (convert_hf_to_gguf.py --mtp) covers DeepSeek-V3.2 and V4 but not the DeepSeek-OCR architecture, and FastMTP's mtp_module.* layout differs from the nextn layers it expects, so the 12 mtp_module.* tensors are dropped at conversion. Output quality is unaffected — the post-trained gains live in the main decoder — but you get plain autoregressive decoding. llama.cpp's own draft-model speculative decoding can be used instead.
• Quantization is applied to the decoder only; the vision tower stays F16.

Verification

Both quants were smoke-tested on a synthetic page containing a heading hierarchy, a paragraph, a 4x4 bordered table and footnotes. Q8_0 and Q4_K_M each returned the complete text with correct heading levels and all twelve table cells accurate, emitting the table as HTML. Q4_K_M was re-tested on 12 Oct 2026 against llama.cpp 0f701c8 (CLI and server) with the same result, and the replacement Q8_0 was tested the same day on two real newspaper scans: body text and figures were transcribed correctly, but on a cropped masthead reading "The Daily W…" both quants completed it as "The Daily Mail", so expect the model to guess text cut off at the image edge. That is a single easy page rather than a benchmark run, so compare against Q8_0 or F16 before relying on Q4_K_M for dense or handwritten documents.

How these were produced

python convert_hf_to_gguf.py jina-ocr-v1/ --outfile jina-ocr-v1-F16.gguf --outtype f16
python convert_hf_to_gguf.py jina-ocr-v1/ --mmproj --outfile mmproj-jina-ocr-v1-F16.gguf --outtype f16
./build/bin/llama-quantize jina-ocr-v1-F16.gguf jina-ocr-v1-Q8_0.gguf   Q8_0
./build/bin/llama-quantize jina-ocr-v1-F16.gguf jina-ocr-v1-Q4_K_M.gguf Q4_K_M

Two adjustments were needed on the conversion side, neither of which affects consumers of these files:

1. The converter has no mapping for the mtp_module.* tensors and aborts with Can not map tensor, so they are filtered out in both the text and mmproj passes. (DeepseekV2Model's built-in MTP skip only matches model.layers.<N> beyond num_hidden_layers, which these tensors don't.)
2. The checkpoint's processor_config.json omits image_mean / image_std, which MmprojModel reads unconditionally. Both are (0.5, 0.5, 0.5), per processing_deepseek_ocr.py.

No imatrix was used.

License and attribution

Original model: jinaai/jina-ocr-v1 by Jina AI, released under CC BY-NC 4.0. These files are a format conversion and quantization of those weights and are redistributed under the same license — non-commercial use only. For commercial licensing, contact Jina AI.

Changes made to the original: conversion from safetensors to GGUF, Q8_0 / Q4_K_M quantization of the decoder, and removal of the FastMTP speculative decoding tensors.

@misc{garcía2026jinaocr,
    title={Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewards},
    author={Alejandro Barón García and Feng Wang and Emilia Garcia Casademont and Han Xiao},
    year={2026},
    eprint={2609.03181},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2609.03181},
}