Ops-Colqwen3-4B-GGUF

Ops-Colqwen3-4B-GGUF

Candle-ready GGUF quantizations of OpenSearch-AI/Ops-Colqwen3-4B.

Ops-Colqwen3-4B is a ColPali-style multimodal embedding model on Qwen3-VL-4B-Instruct. It maps text queries and visual documents (images / PDF pages) into a multi-vector space and scores them with MaxSim.

This repository replaces the original BF16 safetensors with GGUF weights plus a Rust Candle example.

Files

Q4_K_M keeps embeddings, the ColPali projection head, and selected output projections at Q8_0. Remaining 2-D weights are Q4_K. Norms, biases, and other non-2-D tensors stay F32.

GGUF (and Candle 0.11) store at most 4 dimensions. The only 5-D tensor in the original checkpoint is visual.patch_embed.proj.weight, a Conv3d kernel of shape (out, in, t, h, w) = (1024, 3, 2, 16, 16). These files follow llama.cpp's Qwen2-VL / Qwen3-VL converter and split it along t into two F32 Conv2d kernels:

• visual.patch_embed.proj.weight — t = 0, shape (1024, 3, 16, 16)
• visual.patch_embed.proj.weight.1 — t = 1, shape (1024, 3, 16, 16)

Conv3d(t=2) is the sum of those two Conv2d ops, so the convolution is unchanged.

Candle sample

cd examples/candle-ops-colqwen3
cargo run --release -- \
  --model ../../Ops-Colqwen3-4B-Q4_K_M.gguf \
  --tokenizer ../../tokenizer.json \
  --config ../../config.json \
  --dims 128

Default demo:

• images: synthetic pages — a large red circle vs a large yellow triangle
• queries: a large red circle, a large yellow triangle
• output: multi-vector embeddings and a MaxSim score matrix (matching page should rank first)

Custom inputs:

cargo run --release -- \
  --model ../../Ops-Colqwen3-4B-Q4_K_M.gguf \
  --tokenizer ../../tokenizer.json \
  --dims 128 \
  --query "invoice total" \
  --image /path/to/page.png

--dims crops the 2560-d projection to a leading prefix. 128 is the default here; 320 / 640 / 1280 / 2560 remain valid.

Quantization

The quantizer (examples/candle-ops-colqwen3, quantize-gguf) writes the Conv3d split above. Rebuild from the original BF16 shards:

python scripts/quantize_to_gguf.py \
  --src /path/to/Ops-Colqwen3-4B \
  --config config.json \
  --out-dir . \
  --schemes q4_k_m q8_0

Original model

See OpenSearch-AI/Ops-Colqwen3-4B for training details, Vidore numbers, and the Python embedder.

@misc{ops_colqwen3_4b,
  author       = {{OpenSearch-AI}},
  title        = {{Ops-Colqwen3: State-of-the-Art Multimodal Embedding Model for Visual Document Retrieval}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/OpenSearch-AI/Ops-Colqwen3-4B}},
}