Nemotron-3-Embed-1B-BF16-MLX-4bit

Nemotron-3-Embed-1B-BF16-MLX-4bit

An MLX conversion of nvidia/Nemotron-3-Embed-1B-BF16 that runs natively on Apple Silicon. Weights are quantized to MLX affine 4-bit (group_size=64).

This is an independent community conversion. It is not affiliated with or endorsed by NVIDIA. The original model ships vLLM (CUDA) and PyTorch paths only; this repository fills the MLX gap and bundles a self-contained implementation.

The repository name keeps the upstream -BF16 suffix on purpose: NVIDIA publishes both a BF16 and an NVFP4 checkpoint, and this conversion is derived from the BF16 one. -MLX marks the runtime, following the naming convention used outside the mlx-community org.

Changes from the original

The original architecture Ministral3Model is a bidirectional-attention encoder, which mlx-lm / mlx-embeddings do not support out of the box. The following changes were made, and the implementation is bundled as a single self-contained file, nemotron3_embed_mlx.py:

1. Reused attention, yarn RoPE and llama_4_scaling from the ministral3 causal-LM implementation in mlx-lm.
2. Removed the causal mask and replaced it with a key-padding mask, turning the stack into a bidirectional encoder.
3. Applied mean pooling + L2 normalization. Pooling and normalization run in fp32 to avoid bf16 accumulation error perturbing the embedding norm.
4. Applied MLX affine 4-bit quantization (group_size=64).

No weights were retrained. Numerics are unchanged apart from the quantization noted above.

Verification

Checked against the original PyTorch implementation (fp32, identical token ids, six probe texts spanning Korean, English, a 4k-token long-form passage and special characters): worst-case cosine similarity of the pooled embedding 0.975306.

Embeddings are L2-normalized exactly, not approximately: pooling and normalization run in fp32 so bf16 accumulation cannot drift the norm.

Cosine against the reference is not the same measurement as retrieval quality. The 4-bit build sits furthest from the reference on this metric — 0.975 worst case, against 0.9999 for bf16 and 8-bit — and still holds 99.3% of bf16 NDCG@10 across the datasets below. Quantization rotates every vector, and ranking survives as long as they rotate together; the number above says how far this build is from the original, not how much worse it retrieves. Use the retention table for that.

Quantization impact

This repository is a format conversion, not a new model. Absolute retrieval quality belongs to the upstream model — see nvidia/Nemotron-3-Embed-1B-BF16 and NVIDIA's published benchmark figures. This card therefore does not restate absolute quality; it measures the two things the conversion itself can change: fidelity to the original (above) and what quantization costs.

Measured on an Apple M4 (32GB). Every variant saw identical task data on the same machine, which is what makes the comparison valid:

Retention is the per-dataset ratio to bf16, averaged over the nine tasks below.

NanoBEIR — 4 of the 13 tasks, each on its full corpus, so these per-task figures are directly comparable to other published NanoBEIR per-task numbers. The 4-task average is not comparable to a published 13-task mean, so none is given.

NDCG@10 per dataset:

Recall@10 per dataset:

MIRACL — dev-split hard-negatives corpus subsampled to ~6k passages per language (every judged document kept; distractors drawn with a fixed seed, so all variants face the identical task). Subsampling inflates the absolute numbers, so these are for relative comparison between variants only and must not be read against published full-corpus MIRACL results. A second reason not to read them as quality figures: NVIDIA lists MIRACLRetrieval among the upstream model's training datasets, so MIRACL is in-domain for this model, not a zero-shot benchmark. Neither caveat affects the retention comparison, which is what these rows are here for.

NDCG@10 per dataset:

Recall@10 per dataset:

Reproduce with the bundled benchmark_mteb.py: pip install mteb datasets then python benchmark_mteb.py . results.json.

Usage

pip install mlx mlx-lm transformers numpy huggingface_hub

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("choipilkyu/Nemotron-3-Embed-1B-BF16-MLX-4bit")
sys.path.insert(0, path)
from nemotron3_embed_mlx import load, encode

model, tokenizer = load(path)
q = encode(model, tokenizer, ["What is the refund policy?"], input_type="query")
d = encode(model, tokenizer,
           ["Full refunds are available within 14 days of purchase."],
           input_type="passage")
print(float(q[0] @ d[0]))   # embeddings are L2-normalized, so dot == cosine

Prefixes matter. Queries need query: and documents need passage: . The input_type argument applies them for you; pass input_type=None if you have already added the prefix yourself.

Repository contents

The sentence-transformers scaffolding files (modules.json, 1_Pooling/, sentence_bert_config.json, config_sentence_transformers.json) are carried over from the upstream repository as configuration reference. These weights load only via the bundled MLX implementation — not via sentence-transformers or transformers.

Choosing a variant

Measured on an M1 Pro (16GB) over 200 documents averaging 1,014 characters, batch size 8:

Two things follow. At identical precision the MLX path is 1.8x faster than the upstream PyTorch/MPS path. But the quantized variants are ~1.6x slower than bf16, not faster: at 1.1B parameters the dequantization cost outweighs the reduced memory traffic. The same ordering shows up in the Apple M4 runs above (bf16 2.5 docs/s, 8-bit and 4-bit 2.2 docs/s).

Pick bf16 for throughput. Pick 8-bit or 4-bit for footprint — 0.64 GB fits alongside other work on a small machine, and keeps 99.3% NDCG retention.

Reproduce on your own machine with the bundled compare_backends.py.

Limitations

• Default max_length is 4096. The original supports 32k, but bidirectional attention is O(L²) and memory becomes the binding constraint well before that.
• Throughput on Apple Silicon is modest: 2.5 docs/s on long Korean documents (avg 1,014 chars, batch 8) measured on an Apple M4 (32GB). Use a server for bulk indexing and this for development and interactive querying, where single-query latency is not a problem.
• Embeddings from different variants are not interchangeable. Do not mix outputs of different variants in one index.

License

The original model is licensed by NVIDIA under OpenMDW-1.1, and its base model mistralai/Ministral-3-3B-Instruct-2512 under Apache-2.0. Both texts are bundled here as LICENSE and NOTICE.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.