llama-nemotron-rerank-1b-v2-GGUF

llama-nemotron-rerank-1b-v2 GGUF

GGUF quantizations of nvidia/llama-nemotron-rerank-1b-v2 for use with llama.cpp.

Model Details

• Base model: nvidia/llama-nemotron-rerank-1b-v2
• Architecture: Llama 3.2 1B, fine-tuned cross-encoder with bidirectional attention
• Task: Reranking / sequence classification
• Languages: 26 (English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish)
• License: NVIDIA Open Model License + Llama 3.2 Community License

Requirements

These GGUFs require a patched llama.cpp that supports reranking via the llama-embed architecture. Specifically:

1. LLM_TENSOR_CLS and LLM_TENSOR_CLS_OUT in the LLM_ARCH_LLAMA_EMBED tensor allowlist
2. causal_attn and pooling_type hparam reads in the LLM_ARCH_LLAMA_EMBED case

The memoryco/llama.cpp fork (branch pinned-for-rs) includes both patches.

GGUF Fixes Applied

The initial convert_hf_to_gguf.py conversion produced a GGUF that wouldn't load correctly for reranking. These post-conversion fixes were applied:

• Tensor rename: cls.weight → cls.output.weight (maps to LLM_TENSOR_CLS_OUT prefix "cls.output")
• Tensor reshape: [2048] (1D) → {2048, 1} (2D classification head)
• Added metadata: llama-embed.classifier.output_labels: ["relevance"] (sets n_cls_out=1)
• Added metadata: tokenizer.ggml.add_eos_token: true (required for non-zero rerank scores)

Files

Usage with llama.cpp

# Using the patched memoryco/llama.cpp fork
./build/bin/llama-embedding \
  -m llama-nemotron-rerank-1b-v2.Q8_0.gguf \
  --pooling rank \
  -p 'passage: Paris is the capital of France. query: what is the capital of france'

Input format: passage: {document_text}. query: {query_text} as a single string. The model outputs a single float score (higher = more relevant).

Part of the MemoryCo Nemotron Stack

• Generation: memoryco-ai/Nemotron-3-Nano-4B-GGUF
• Embeddings: memoryco-ai/llama-nemotron-embed-1b-v2-GGUF
• Reranking: This repo

Converted by memoryco-ai using llama.cpp's convert_hf_to_gguf.py with custom architecture support and post-conversion GGUF fixes.