olmOCR-7B-0825-FP8-GGUF

olmOCR-7B-0825-FP8 GGUF

GGUF conversion of allenai/olmOCR-7B-0825-FP8, published by Mu2 Solutions. First GGUF release of this model.

olmOCR-7B is Allen Institute for AI's document-understanding model — reads and reasons about text in images (PDFs, screenshots, scanned documents). Built on the Qwen2.5-VL architecture, so it runs on standard llama.cpp multimodal tooling with no custom forks required.

License

Apache 2.0 — same as the source model.

Files

Q4_K_M is the recommended default. Pair any text quant with the mmproj for document/vision use.

Conversion

• Source: HF safetensors (3 shards) via the llama.cpp HF-to-GGUF converter
• Architecture: qwen2_5_vl (official llama.cpp support — text + mmproj both standard)
• mmproj: 519 tensors (Qwen2.5-VL vision tower)
• Chat template embedded from source

Verification

• Text: verified — clean decoder output.
• Vision: verified — image encoding completes correctly and generation produces coherent descriptions ("The image shows...") through the standard llama-mtmd-cli tool. Full OCR run on large screenshots is CPU-slow but functional; GPU offload recommended for production use (-ngl 99).

Usage

# Document OCR / vision chat (standard llama.cpp)
llama-mtmd-cli -m olmOCR-7B-0825-FP8-text-Q4_K_M.gguf \
  --mmproj olmOCR-7B-0825-FP8-mmproj-F16.gguf \
  --image document.png -p "Read the text in this image."

# OpenAI-compatible serving
llama-server -m olmOCR-7B-0825-FP8-text-Q4_K_M.gguf \
  --mmproj olmOCR-7B-0825-FP8-mmproj-F16.gguf --jinja

Credit

• Model: Allen Institute for AI (Ai2) — Apache 2.0
• Conversion: Mu2 Solutions — promoting truly open-source AI, one GGUF at a time