SauerkrautLM-Multi-ColBERT-33m-GGUF

SauerkrautLM-Multi-ColBERT-33m-GGUF (ColBERT GGUF)

This repository contains GGUF format weights for the ColBERT retrieval model VAGOsolutions/SauerkrautLM-Multi-ColBERT-33m, converted specifically for our GGML-based ColBERT C++ runtime.

This GGUF file is NOT a standard llama.cpp decoder model. It uses the pg_colbert_v1 layout containing backbone transformer weights, tokenizer metadata, ColBERT dense projection layers, and similarity metrics.

Model Summary

• Source Model: VAGOsolutions/SauerkrautLM-Multi-ColBERT-33m
• Format: GGUF (pg_colbert_v1 schema)
• Precision: F16
• Encoder Backbone: ModernBERT / BERT
• ColBERT Projection Dimension: 128
• Profile Schema: pg_colbert_profile_v1
• Target Runtime: both

Parity & Verification Report

• Strict PyLate Token-Plan Parity: PASSED
• Vector Parity Checked: YES
• Strict Vector Parity Status: PASSED
• Vector Parity Claim: PASSED (verified numerical equivalence with reference PyLate embeddings)

Usage with ColBERT GGML C++ Runtime

This GGUF model is designed to be loaded directly by our custom C++ ColBERT runtime (which uses GGML for the ModernBERT/BERT backbone forward pass and executes the ColBERT late-interaction dense projection).

Loading the Model in C++

// Initialize the ColBERT GGML model context
colbert_model model = colbert_model_load("SauerkrautLM-Multi-ColBERT-33m.f16.gguf");

// Tokenize and encode queries into late-interaction token embeddings
std::vector<float> query_embeddings = colbert_encode_query(model, "Which planet is known as the Red Planet?");

// Tokenize and encode documents into late-interaction token embeddings
std::vector<float> doc_embeddings = colbert_encode_doc(model, "Mars is the Red Planet.");

GGUF Conversion Info

Generated using the convert_colbert_hf_to_gguf.py utility.

• CLI Command: python tools/convert_colbert_hf_to_gguf.py --model-id VAGOsolutions/SauerkrautLM-Multi-ColBERT-33m --outfile VAGOsolutions_SauerkrautLM_Multi_ColBERT_33m.f16.gguf --outtype f16 --target-runtime both
• Converter Version: 1.0.0