llama-nemotron-rerank-1b-v2-ONNX

llama-nemotron-rerank-1b-v2 — ONNX Export

ONNX export of nvidia/llama-nemotron-rerank-1b-v2 for CPU/GPU inference via ONNX Runtime and fastembed-rs.

Available variants

All variants produce identical logits [batch, 1] output and pass the relevance sanity check (panda/relevant doc scores significantly higher than irrelevant).

Usage with fastembed-rs

[dependencies]
fastembed = "5"

use fastembed::{TextRerank, RerankInitOptions, RerankerModel};

let mut model = TextRerank::try_new(
    RerankInitOptions::new(RerankerModel::LlamaNemotronRerank1BV2Int4Full)
        .with_show_download_progress(true),
)?;

let results = model.rerank(
    "what is a panda?",
    vec![
        "A panda is a large black-and-white bear native to China.",
        "The sky is blue and the grass is green.",
    ],
    true,
    None,
)?;
// results[0].score ≈ 1.50, results[1].score ≈ -6.87

Available RerankerModel variants:

• RerankerModel::LlamaNemotronRerank1BV2 — FP32
• RerankerModel::LlamaNemotronRerank1BV2Int8 — INT8
• RerankerModel::LlamaNemotronRerank1BV2Int4Full — INT4+INT8-emb (recommended)

Usage with ONNX Runtime (Python)

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

session = ort.InferenceSession(
    "model_int4_full.onnx",
    providers=["CPUExecutionProvider"],
)
tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/llama-nemotron-rerank-1b-v2", trust_remote_code=True
)

query = "what is a panda?"
documents = [
    "A panda is a large black-and-white bear native to China.",
    "The sky is blue and the grass is green.",
]

enc = tokenizer(
    [query] * len(documents),
    documents,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="np",
)
logits = session.run(
    ["logits"],
    {
        "input_ids": enc["input_ids"].astype(np.int64),
        "attention_mask": enc["attention_mask"].astype(np.int64),
    },
)[0]
scores = logits[:, 0].tolist()
ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
    print(f"{score:7.3f}  {doc[:80]}")

For the FP32 and INT4 variants with external data, pass the .onnx file path — ONNX Runtime resolves the .onnx_data sidecar automatically from the same directory.

Export details

• Base model: nvidia/llama-nemotron-rerank-1b-v2 (LLaMA-3.2-1B with bidirectional attention, sequence-classification head)
• Exported with: optimum 1.23.3 + transformers 4.48.3, opset 17, TorchScript path (dynamo=False)
• Custom OnnxConfig required due to LlamaBidirectionalForSequenceClassification architecture (trust_remote_code=True)
• INT8: onnxruntime.quantization.quantize_dynamic, weight_type=QInt8, MatMulConstBOnly=True
• INT4Full: MatMulNBitsQuantizer(bits=4, block_size=32, is_symmetric=True) applied first, then quantize_dynamic for Gather (embedding) INT8

---

Model Overview

The Llama Nemotron Reranking 1B model is optimized for providing a logit score representing how relevant a document is to a given query. Fine-tuned for multilingual, cross-lingual text question-answering retrieval, with support for long documents (up to 8192 tokens). Evaluated on 26 languages.

Architecture

Architecture Type: Transformer Network Architecture: Fine-tuned ranker from meta-llama/Llama-3.2-1B with bidirectional attention.

The model is a transformer cross-encoder fine-tuned with contrastive learning. Bidirectional attention is applied during fine-tuning for higher accuracy. Mean pooling over the last decoder output is used with a binary classification head for ranking.

Inputs / Outputs

• Input: input_ids [batch, seq_len], attention_mask [batch, seq_len]
• Output: logits [batch, 1] — raw relevance score per (query, document) pair. Apply sigmoid for probabilities.

Evaluation Results

License

Use of this model is governed by the NVIDIA Open Model License Agreement. Additional Information: Llama 3.2 Community Model License Agreement.

Provenance and EU AI Act Art. 53 note

• Upstream model: nvidia/llama-nemotron-rerank-1b-v2 — published by nvidia.
• Upstream licence: other. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (ONNX). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.