granite-embedding-311m-multilingual-r2-GGUF

Granite-Embedding-311M-Multilingual-R2 (GGUF)

Granite-Embedding-311M-Multilingual-R2 is a multilingual embedding model developed by IBM Granite for semantic search, retrieval, retrieval-augmented generation (RAG), similarity matching, and cross-lingual information retrieval workflows. This repository contains a Q6_K GGUF quantized version of the model optimized for efficient local embedding generation.

Unlike generative language models, this model is designed to convert text, queries, passages, documents, and code into dense vector embeddings that capture semantic meaning. These embeddings can be used for similarity search, ranking, clustering, and retrieval applications.

The Q6_K quantized format reduces memory requirements while preserving strong embedding quality, making it suitable for production retrieval systems, vector databases, and enterprise-scale search applications.

---

Model Overview

• Model Name: Granite-Embedding-311M-Multilingual-R2
• Base Model: ibm-granite/granite-embedding-311m-multilingual-r2
• Architecture: ModernBERT-based Bi-Encoder
• Parameter Count: 311 Million
• Context Window: 32,768 Tokens
• Embedding Dimension: 768
• Modalities: Text
• Language Support: 200+ Languages
• Developer: IBM Granite
• License: Apache 2.0

---

Training Background (Original Model)

Granite-Embedding-311M-Multilingual-R2 is trained for multilingual dense retrieval and semantic representation learning across a broad range of languages and domains.

Pretraining

• Built on a ModernBERT-based encoder architecture
• Trained across multilingual corpora, retrieval datasets, and code-related datasets
• Focused on semantic representation learning and retrieval effectiveness
• Optimized for multilingual and cross-lingual retrieval tasks

Retrieval Optimization

• Enhanced using contrastive fine-tuning techniques
• Improved through knowledge distillation and model merging strategies
• Optimized for query-passage alignment and semantic similarity matching
• Designed to improve retrieval quality across multilingual and enterprise search workloads

---

Quantization Formats

This repository provides GGUF quantized versions of the Granite-Embedding-311M-Multilingual-R2 model optimized for efficient embedding generation and retrieval workloads.

IQ4_NL

• Size reduction of approx 61.90% (232 MB) compared to 16-bit (609 MB)
• Advanced 4-bit non-linear quantization designed to better preserve embedding quality and retrieval performance
• Better suited for semantic search, multilingual retrieval, and vector database applications
• Generally provides stronger embedding consistency and reduced quantization loss compared to more aggressive quantization formats
• Slightly increased computational overhead during embedding generation

IQ4_XS

• Size reduction of approx 62.40% (229 MB) compared to 16-bit (609 MB)
• Balanced 4-bit quantization focused on efficient embedding generation and stable retrieval performance
• Provides a practical balance between memory efficiency, retrieval quality, and inference speed
• Suitable for semantic search, vector databases, document retrieval, and similarity-search applications
• Maintains reliable embedding quality across most practical deployment scenarios

Q6_K

• Size reduction of approx 56.81% (263 MB) compared to 16-bit (609 MB)
• Higher-precision 6-bit K-Quant format designed to preserve embedding fidelity and retrieval quality
• Better suited for production-grade semantic search, multilingual retrieval, and RAG pipelines
• Provides stronger similarity matching performance and reduced quantization loss compared to lower-bit formats
• Requires more memory than 4-bit variants but typically delivers higher retrieval accuracy and embedding consistency

---

Key Capabilities

• Semantic Search Generates embeddings suitable for high-quality semantic search systems.

• Retrieval-Augmented Generation (RAG) Supports document retrieval pipelines for RAG applications.

• Multilingual Retrieval Enables retrieval workflows across more than 200 languages.

• Cross-Lingual Search Supports semantic matching between different languages.

• Code Retrieval Generates embeddings suitable for code search and retrieval applications.

• Long-Context Retrieval Supports embedding generation for large documents with up to 32K token context windows.

---

Recommended Usecases

• Semantic Search Systems Build vector-search and semantic retrieval applications.

• Retrieval-Augmented Generation (RAG) Improve document retrieval quality for AI assistants and enterprise knowledge systems.

• Multilingual Information Retrieval Search across multilingual document collections using shared semantic representations.

• Vector Database Pipelines Generate embeddings for storage and retrieval in vector databases.

• Code Search Support code retrieval and developer-assistance workflows.

• Enterprise Knowledge Retrieval Enable large-scale search across internal documents and structured knowledge repositories.

---

Acknowledgments

These quantized models are based on the original work by the IBM Granite development team.

Special thanks to:

• The IBM Granite team for developing and releasing the Granite-Embedding-311M-Multilingual-R2 model.

• Georgi Gerganov and the llama.cpp open-source community for enabling efficient GGUF quantization and local inference.

---

Contact

For questions, feedback, or support, please reach out at [email protected] or visit https://www.sandlogic.com/