GOT-OCR2_0-4bit

mlx-community/GOT-OCR2_0-4bit

4-bit MLX quantization of stepfun-ai/GOT-OCR2_0, the 560M-parameter OCR-2.0 model. Converted with mlx-vlm 0.6.14 (mlx 0.32.0) for inference on Apple Silicon.

Fidelity was measured against the unquantized bf16 source and against the reference PyTorch implementation, not assumed. Every number below comes from a run recorded in this card.

Requires mlx-vlm with GOT-OCR 2.0 support, which is currently open as Blaizzy/mlx-vlm#1908. The PR is not merged yet, so a released mlx-vlm cannot load this repo.

python -m mlx_vlm generate \
    --model mlx-community/GOT-OCR2_0-4bit \
    --image document.png \
    --prompt "OCR: " \
    --max-tokens 1024

GOT is not a chat model. It takes two instructions: OCR: for plain text and OCR with format: for structured output (tables, formulas, sheet music). Any other prompt is out of distribution.

Quantization

The effective figure sits well above the nominal 4 because mlx-vlm quantizes only the language model. The vision tower and the projector are 96.7M of the 560.5M parameters, 17%, and stay bf16 by design. Packing checks out exactly: U32 x (32/4) = 463,785,984, the quantized parameter count.

Tied embeddings are 155.5M of those 463.8M, 34%, much higher than in a typical conversion where the transformer body dominates. They are the worst-hit tensor in both quantizations, which is what the per-tensor numbers below show.

Fidelity

Weight level, against the bf16 source

Exact, prompt-free and hardware-independent: the quantized weights are dequantized and compared element-wise against the source.

Against the reference PyTorch implementation

One forward pass, same image and same prompt (the MPT conversation the model was trained with), comparing the final-position logits over all 151,860 classes.

The second row is the one that matters: the source weights are bf16, so running the MLX port in fp32 loses nothing, and the residual is porting error alone. The first row's larger figure is bf16 rounding, not a defect.

Task level, against ground truth

Six documents rendered locally with exactly known text (invoice, lab report, shipping label, receipt, spec table, rotated receipt), transcribed with OCR: and scored directly. This sidesteps using bf16 as the reference at all.

field counts required strings present exactly, content ignores markup and whitespace, numeric counts ground-truth numbers recovered.

The 8-bit output is byte-identical to bf16 on all six documents, down to the same fields missed.

Two caveats a reader should have:

• The 4-bit field score is higher than bf16 and that is not an improvement. The denominator is 38 strings, so one string is 0.026. 4-bit happens to get the invoice fully right and then loses 12.00 on the receipt. content and numeric, which are less format-sensitive, both move the other way. Treat the three variants as separated by one or two strings, not by field.
• shipping_label loses the same three fields in every variant, bf16 included (1Z-994-AX-77310582, Halberd Components, 3011AB). That is a limit of the source model on that document, not a quantization effect.

Throughput measured on an M-series Mac during the same run, single image, generation only.

What was not measured

• Only the plain 1024x1024 single-crop path. GOT's fine-grained mode (region by box or by colour), its multi-crop path for dense pages, and its multi-page mode are not exercised by anything above.
• OCR with format: was not scored. The accuracy table uses OCR: only. Format mode was smoke-tested and runs, but no ground-truth scoring was done for tables, formulas or sheet music.
• No perplexity, top-1 agreement or KL. GOT only emits transcriptions, so generic passages give meaningless perplexity. The ground-truth OCR scoring above replaces that layer rather than supplementing it.
• Six synthetic documents, one language. They are rendered, not photographed, and English only. Real scans, handwriting and the multilingual claims of the source are untested here.
• No accuracy measurement at long context. Everything ran well under the 32k window.

Related

• mlx-community/GOT-OCR2_0-bf16
• mlx-community/GOT-OCR2_0-8bit
• mlx-community/GOT-OCR2_0-4bit

Source model: stepfun-ai/GOT-OCR2_0 (paper).