typhoon-ocr1.5-2b-gguf

GGUF Model from typhoon-ai/typhoon-ocr1.5-2b and mmproj (multimodal projection) file from Qwen/Qwen3-VL-2B-Instruct-GGUF since mmproj are not modified from typhoon-ocr finetune.

These files are compatible with llama.cpp, Ollama, and other GGUF-based tools, supporting inference on CPU, NVIDIA GPU (CUDA), Apple Silicon (Metal), Intel GPUs (SYCL), and more. You can mix precision levels for the language and vision components based on your hardware and performance needs, and even perform custom quantization starting from the FP16 weights.

How to Use

To use these models with llama.cpp, please ensure you are using the latest version—either by building from source or downloading the most recent release according to the devices.

You can run inference via the command line or through a web-based chat interface.

CLI Inference (llama-mtmd-cli) For example, to run Qwen3-VL-2B-Instruct with an FP16 vision encoder and Q8_0 quantized LLM:

llama-mtmd-cli \
-m path/to/typhoon-ocr1.5-2b.gguf \
--mmproj path/to/mmproj-typhoon-ocr1.5-2b-F16.gguf \
--image test.jpeg \
-p "Below is an image of a document page. Your task is to return the markdown representation of this document, presenting tables in HTML format as they naturally appear." \
--temp 0.7 --top-k 20 --top-p 0.8 -n 1024

Web Chat (using llama-server) To serve typhoon-ocr1.5-2b via an OpenAI-compatible API with a web UI:

llama-server \
-m path/to/typhoon-ocr1.5-2b.gguf \
--mmproj path/to/mmproj-typhoon-ocr1.5-2b-Q8_0.gguf

Once the server is running, open your browser to http://localhost:8080 to access the built-in chat interface, or send requests to the /v1/chat/completions endpoint. For more details, refer to the official documentation.