Nemotron-3-Embed-1B-CoreML-S256-W4A16

Nemotron-3-Embed-1B Core ML S256 W4A16

<!-- polished-overview:start -->

Overview

Nemotron-3-Embed-1B-CoreML-S256-W4A16 is a Core ML conversion intended for deployment on Apple platforms, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.bin (1 file)
• Additional configuration, tokenizer, processor, or shard files (13 visible artifacts total)

Quick start

Core ML

Use the compiled Core ML assets from an Apple-platform application or a current Core ML tooling stack. Check the repository's input/output metadata before wiring the model into a production pipeline.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

This repository contains an independently converted Core ML version of nvidia/Nemotron-3-Embed-1B-BF16. It uses symmetric INT4 weight-only quantization with block size 32. Activations remain floating point (W4A16). No weights were retrained.

The exported contract is fixed-shape:

• input_ids: int32[1, 256]
• attention_mask: int32[1, 256]
• embedding: normalized 2,048-dimensional vector
• left padding
• query: prefix for queries
• passage: prefix for documents
• masked mean pooling followed by L2 normalization

This is a community conversion. It is not affiliated with or endorsed by NVIDIA.

Accuracy summary

Conversion fidelity

The conversion used the immutable upstream revision a5e0f804b9e90a1ca6784ecbf6e41595774fc834.

These are mechanical single-input parity checks. They verify the conversion path but do not replace retrieval evaluation.

Bilingual conversation-retrieval test

We also evaluated the INT4 model on a deliberately difficult synthetic set of 16 English/Spanish conversation summaries and 16 queries. The cases include corrections, negation, superseded facts, and near-topic distractors. The Core ML model and PyTorch reference received identical token IDs, prefixes, left padding, sequence length, and cosine-ranking methodology.

Top-1 agreement and vector cosine compare INT4 directly with the PyTorch reference. Lower-dimensional rows slice the first N dimensions and renormalize before cosine search.

The apparent INT4 improvement at 768 and 2,048 dimensions is one changed rank on a very small set. It is quantization noise, not evidence that INT4 is more accurate than the reference model.

Findings

• INT4 preserved the reference model's top-1 result for every query at 256 and 512 dimensions.
• At 256 dimensions, all retrieval metrics matched the reference exactly even though the minimum per-vector cosine was 0.91556.
• Fidelity increased as more output dimensions were retained: mean cosine rose from 0.96609 at 256 dimensions to 0.98236 at 2,048 dimensions.
• The test is useful for quantization parity on conversational corrections and distractors, but it is synthetic and small. It must not be interpreted as an MTEB score or a general retrieval-quality estimate.
• No claim is made here about tasks, languages, or sequence lengths outside the included evaluation.

Raw results are available in results/conversion-report.json and results/conversation-retrieval-benchmark.json.

Usage

from pathlib import Path

import coremltools as ct
import numpy as np
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

root = Path(snapshot_download("groxaxo/Nemotron-3-Embed-1B-CoreML-S256-W4A16"))
tokenizer = AutoTokenizer.from_pretrained(
    root / "tokenizer",
    local_files_only=True,
    padding_side="left",
)
model = ct.models.MLModel(
    str(root / "model" / "Nemotron3Embed1B-S256-W4A16-B32.mlpackage")
)

encoded = tokenizer.encode("query: What decision was corrected?", add_special_tokens=True)
encoded = encoded[:256]
pad_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 11
padding = 256 - len(encoded)

input_ids = np.asarray([[pad_id] * padding + encoded], dtype=np.int32)
attention_mask = np.asarray([[0] * padding + [1] * len(encoded)], dtype=np.int32)

embedding = model.predict(
    {"input_ids": input_ids, "attention_mask": attention_mask}
)["embedding"].reshape(-1)
embedding /= max(float(np.linalg.norm(embedding)), 1e-12)
print(embedding.shape)  # (2048,)

Reproduction

The conversion and evaluation scripts are included under scripts/. The recorded environment used Python 3.12, PyTorch 2.7.1, Core ML Tools 9.0, and Transformers 5.5.0.

python scripts/convert_nemotron_coreml.py \
  --output-dir artifacts \
  --sequence-length 256 \
  --revision a5e0f804b9e90a1ca6784ecbf6e41595774fc834

python scripts/benchmark_nemotron_conversations.py \
  --mlmodel artifacts/Nemotron3Embed1B-S256-W4A16-B32.mlpackage \
  --tokenizer artifacts/tokenizer \
  --sequence-length 256 \
  --output results/conversation-retrieval-benchmark.json

The benchmark script compares against the exact upstream BF16 revision and expects those upstream weights to be available in the local Hugging Face cache.

License and attribution

This repository includes the upstream LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md. Use is subject to those terms.