Nemotron-3-Embed-8B-Community-FP8

<a href="https://shadowrock.io"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-white.svg"> <img alt="ShadowRock" src="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-black.svg" width="340"> </picture> </a>

Nemotron-3-Embed-8B — Community FP8

Unofficial community quantization — not an NVIDIA release.

FP8 (E4M3) build of nvidia/Nemotron-3-Embed-8B-BF16 (revision 8ca3ff38), the top-ranked open embedding model on the RTEB leaderboard at time of writing. All credit for the base model belongs to NVIDIA; this repo only changes the weight storage format. ~8.5 GB of weights (half of BF16), native FP8 execution on Ada, Hopper, and Blackwell GPUs, and retrieval quality that ties the unquantized model within run noise: every task we measured lands within 0.008 nDCG@10 of NVIDIA's own published numbers, and the long-document task within 0.0001 of our BF16 baseline.

This is the variant to pick when you want maximum quality at half the memory. The companion NVFP4 build trades a little more fidelity for ~3.5× compression; the MLX 4-bit build serves Apple-Silicon Macs.

Benchmarks vs the unquantized model

Comparison column = NVIDIA's official per-task results from the mteb results repo — their numbers, not our reproduction. Our runs: mteb 2.18.12, vLLM 0.26.0, mean pooling, query: /passage: prefixes, max length 8192 (NVIDIA evaluated at 4096; on these tasks few documents exceed either limit). Tasks are the open (public) RTEB datasets in the domains where the base model ranks top-4 on the RTEB leaderboard: Finance #1, German #1, Code #2, Healthcare #4, Legal #4.

Mean delta −0.0019 across all 12 tasks; −0.0013 on the 10-task subset shared by all three community builds (the two AILA legal tasks were run only on the CUDA builds). The private RTEB datasets can only be run by the MTEB team, so this table covers the open subset.

Regression vs our own BF16 baseline (identical harness both sides)

BF16 baseline computed with the same code, adapter, prefixes, and pins on an A100. Gate: per-task nDCG@10 loss ≤ 0.01.

All pass. Embedding-level fidelity vs BF16 on token-ID-locked fixtures: cosine 0.9967–0.9979 (runtime kernels), matching the ModelOpt fake-quant simulation (0.9982–0.9990). Raw result JSON ships under results/.

Serving with vLLM

from vllm import LLM
from vllm.config import PoolerConfig

llm = LLM(
    model="shadowrock-io/Nemotron-3-Embed-8B-Community-FP8",
    runner="pooling",
    pooler_config=PoolerConfig(seq_pooling_type="MEAN"),  # default LAST is silently wrong
    max_model_len=8192,
)
out = llm.embed(["query: what does FP8 change?", "passage: Only the weight format."])

Required patch for vLLM ≤ 0.26.0: vLLM's pooling adapter replaces the checkpoint's absent lm_head with a placeholder layer, and ModelOptFp8LinearMethod.process_weights_after_loading crashes on the placeholder's meta tensors (Tensor.item() cannot be called on meta tensors). Run scripts/patch_modelopt_fp8_guard.py once against your vLLM install before loading (idempotent; an upstream fix has been proposed).

Notes that matter for correct embeddings:

• Pooling must be MEAN and attention is bidirectional; both come from the checkpoint config, but the pooler override above guards against defaults.
• Prefixes are your job: query: / passage: . The server does not add them.
• Texts longer than max_model_len are rejected by vLLM's pooling runner — truncate at the tokenizer (truncation=True, max_length=8192) and pass token IDs.
• Embeddings are 4096-dim; L2-normalize before use. Matryoshka truncation (2048/1024): slice, then re-normalize.
• On pre-Ada GPUs (SM < 89, e.g. A100) vLLM falls back to weight-only Marlin kernels — functional, but not the W8A8 path measured here.

Measured on: NVIDIA H200 (validation runs), GeForce RTX 5070 Ti (SM120, serving validation), vLLM 0.26.0, CUDA 12.8.

Quantization details

• Method: NVIDIA TensorRT Model Optimizer (ModelOpt) FP8 post-training quantization — E4M3 weights with per-tensor activation scales; embeddings, norms, and pooling untouched. Full module inventory: quantization/module_inventory.json.
• Derived in a fresh process from the pinned BF16 snapshot — never from another quantized model object.
• Calibration: ~1k public samples from MS MARCO and MIRACL train splits, token-bucketed (32–16k tokens) with real prefix distribution. MS MARCO is research-licensed, so the manifest ships dataset IDs + a deterministic builder script, not text. Eval-set contamination audit (by ID and content hash) included.
• Quantize/eval scripts ship under scripts/; raw eval JSON under results/.

Caveats

• MIRACL multilingual coverage in the regression suite is two held-out languages (Swahili, Telugu, hard-negatives variants) on the NVFP4 companion; this FP8 build's multilingual evidence is LegalQuAD (German) plus the base model's own multilingual results. Full-corpus MIRACL was excluded for compute cost.
• NVIDIA evaluated at sequence length 4096; our runs use 8192. On the tasks above the difference is immaterial (few documents exceed 4096 tokens), but it is a protocol difference.

Intended use & limitations

Intended uses are the base model's: dense retrieval, semantic search, and RAG indexing over text corpora, with query: /passage: prefixed inputs. The base card's intended-use, safety, and language-coverage statements — nvidia/Nemotron-3-Embed-8B-BF16 — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the benchmarks listed above and nothing beyond them: other languages, domains, sequence-length regimes, and hardware/runtime combinations inherit the base model's behavior with quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the NVIDIA base model — the embedding quality is theirs:

@misc{nvidia2026nemotron3embed,
  title  = {Nemotron-3-Embed-8B},
  author = {NVIDIA},
  year   = {2026},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16}
}

License

OpenMDW-1.1, inherited from the base model (see LICENSE). NOTICE carries the upstream Apache-2.0 attribution for the Ministral component plus our modification statement. Community build by ShadowRock; no NVIDIA affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.