Nemotron-Parse-2.0-4bit

Nemotron-Parse-2.0-4bit

4-bit MLX quantization of nvidia/NVIDIA-Nemotron-Parse-2.0, a 903M-parameter encoder-decoder OCR model (C-RADIO vision encoder + mBART-style decoder). Converted with mlx-vlm 0.6.12 (mlx 0.32.0) for inference on Apple Silicon.

This is the first MLX conversion of the Nemotron-Parse architecture. The model is image-to-text only: the prompt text is ignored, the image alone drives the transcription (this matches the source implementation).

Fidelity was measured against the source, not assumed. The numbers below and the method that produced them are in the tables that follow.

Loading requires the nemotron_parse architecture port, new in mlx-vlm (PR #1866). Until it merges, install the port branch: pip install git+https://github.com/axiom-of-choice/mlx-vlm@feat/nemotron-parse

pip install mlx-vlm
python -m mlx_vlm generate \
    --model mlx-community/Nemotron-Parse-2.0-4bit \
    --image document.png \
    --prompt "Extract the text" \
    --max-tokens 512

Quantization

Effective bits per weight exceeds the nominal 4 because mlx-vlm quantizes only the language side and leaves the C-RADIO vision encoder in bf16 by design. The vision tower is 72% of all parameters, which is why the effective figure is much higher than on a decoder-only model. Verified, not assumed: of 766 unquantized tensors, zero carry a .scales entry.

Measured fidelity

Weight-level metrics compare dequantized weights against the source tensor by tensor, streamed from disk (the source ships fp32; the 8-bit SNR band for a bf16-comparable baseline is 42.6–42.8 dB). No prompts, no sampling, hardware-independent.

Measured on an M2 Pro / 32 GB, greedy decoding.

OCR accuracy vs ground truth

Because OCR has ground truth, transcription can be scored directly rather than inferred from token distributions. Six documents were rendered locally with exactly known content (invoice, clinical lab report, shipping label, receipt, motor-controller datasheet, rotated receipt).

• Field accuracy: required strings present, exact match (format-sensitive).
• Content accuracy: the same, ignoring markup and whitespace.
• Numeric recall: ground-truth numbers present, ignoring markup.

Per-document field accuracy:

The 4-bit spec_table regression

At 4 bits the datasheet document transcribes as a degenerate & \ repetition loop (field accuracy 0.00 vs 0.87 at 8 bits). The argmax flips at the second decoding step and the loop is self-reinforcing; NVIDIA's own generation config (repetition_penalty 1.1, applied in this harness) does not break it. This is a real quantization trade at 4 bits on dense table layouts — the same document class where the model's native LaTeX output is token-heavy. Every other document scores 1.00 at 4 bits.

The greedy path is otherwise near-tie sensitive: on lab_report the 4-bit variant transcribes all fields where bf16 and 8-bit miss two (a 0.00-vs-0.05 logit margin flips the path), so per-document ordering between 4/8/bf16 is not meaningful; the aggregate tables are.

What was not measured

• No distributional metrics (perplexity, top-1 agreement, KL). The model is image-to-text only — text prompts are ignored — so a text-passage harness cannot score it.
• No standard OCR benchmark. No OmniDocBench, no DocVQA. The six documents are synthetic and rendered locally: exact ground truth, but not photographs, not handwriting, not skewed scans, and not non-Latin scripts.
• No judged quality, no layout-structure scoring, no throughput figures for any chip other than an M2 Pro.
• The source's own golden generation sequence is hardware-sensitive. The model's greedy path diverges between CUDA bf16 (NVIDIA's golden file) and CPU/Metal bf16 at the third token; the MLX port reproduces the CPU reference byte-for-byte. Verified against the Hugging Face implementation on the same inputs.

Sibling variants

Credits

All credit for the underlying model belongs to NVIDIA. This is a format conversion and quantization; no training or fine-tuning was performed. The architecture port (mlx_vlm/models/nemotron_parse/) is new in mlx-vlm.