unlimited-ocr-4bit-mixed-mlx

Unlimited-OCR — 4bit-mixed MLX

Mixed precision: 4-bit experts/MLP, 8-bit attention projections, embeddings and LM head; vision encoders (SAM + CLIP) and the vision→language projector kept unquantized (K-quant philosophy applied to MLX). Converted from baidu/Unlimited-OCR (MIT) with mlx-vlm 0.6.3 on 2026-07-01.

Part of a measured-ladder series for this model; conversions across runtimes evaluated with the same harness are in the collection.

Smallest variant that mostly works, but its quality is uneven: strong on clean prose and digit-heavy invoices (0.00% and 0.00% CER), weak on dense small-font pages (62.58% CER, including 1 looping page). Its 20.86% overall CER is dominated by that 1 looping page; on the other 23 pages it averages ~2.08% CER. Trade-offs: that page still loops, and heterogeneous quantization decodes slower (138 tok/s, below the 8-bit variant). If you can afford 3.19 GB, the 6-bit variant is the recommended default.

This release measures its quantization cost against exact rendered ground truth, so the numbers are interpretable: every figure below traces to a reproducible evaluation run. Every variant in this series was evaluated against the BF16 baseline on the same reproducible corpus before publishing.

Which variant should I use?

Recommended default: the 6-bit variant — no measured degradation on this corpus (CER within noise of the BF16 baseline) at 3.19 GB. The table below is the same in every card in this series; pick by your size budget.

BF16 is the reference baseline (overall CER 1.62% on this corpus).

Measured quality (vs BF16 baseline)

Deltas are computed on unrounded values, so displayed cells may differ by 0.01pp.

Loop rate counts pages whose output ran longer than 1.5× the reference — the runaway-repetition failure mode. CER excl. loop pages is the mean over the remaining pages, so one collapsed page is visible as a loop rather than silently inflating the headline CER.

Corpus: 24 synthetic pages (3 difficulty tiers × 8), deterministic seed 20260701, exact rendered ground truth, temperature 0, prompt document parsing. (Free OCR. degenerates into a repetition loop on this mlx-vlm build, so document parsing. is used for every variant). Decoding runs with no repetition penalty, deliberately, to expose quantization instability as honest high CER; the official inference pipeline applies no_repeat_ngram_size=35, which suppresses exactly the runaway repetition these loop pages capture, so deployed output on flagged pages will not collapse the way these numbers show. Normalization (grounding tokens, markdown decoration, whitespace) applied identically to reference and hypothesis, so CER reflects recognition, not formatting. Reference text is known character-for-character, and per-page hypothesis/reference length ratios back the loop rate above. OCR ladders scored against real-form datasets with partial annotations can produce CER above 1.0, where degradation-per-bit stops being readable; the corpus design here exists to avoid that. The parsing prompt emits structural region labels as <|det|>label [x,y,x,y]<|/det|> preambles; these are stripped by the same normalization that removes grounding tokens and coordinates (a scoped rule that matches only that exact structure, never body text), so absolute CER reflects text recognition rather than layout markup — both the absolute figures and the degradation-vs-BF16 deltas are meaningful. Eval harness (corpus generator + CER/WER scorer) is open-source: https://github.com/vimalnakrani08/unlimited-ocr-eval-harness (the scoring core is backend-agnostic; the runner is MLX-specific).

Where the recovery comes from

An ablation isolating the projector (float projector, all else identical to plain 4-bit) still loops on 7/24 pages at 115% overall CER — projector precision shifts which pages tip into loops (trading a numeric-page loop for a dense-page loop) but not how many. The recovery in this variant is therefore decoder-side: 8-bit attention projections, embeddings, and LM head.

Usage

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("vimalnakrani/unlimited-ocr-4bit-mixed-mlx")
config = load_config("vimalnakrani/unlimited-ocr-4bit-mixed-mlx")
prompt = apply_chat_template(processor, config, "document parsing.", num_images=1)
print(generate(model, processor, prompt, image="page.png",
               max_tokens=2600, temperature=0.0).text)

Prompt vocabulary follows DeepSeek-OCR: document parsing. for markdown/layout, Free OCR. for plain text, prefix <|grounding|> for bounding boxes. These measurements use document parsing.; on this mlx-vlm build Free OCR. was observed to degenerate into a repetition loop, so prefer document parsing. here. Note also that mlx-vlm 0.6.3 loads this model's tokenizer as a slow LlamaTokenizer that does not apply byte-level BPE decoding, so raw generate().text may contain byte markers (spaces as Ġ, newlines as Ċ); decode them before use (the eval harness does this).

How this was made

1. config.json: model_type re-badged unlimited-ocr → deepseekocr (tensor layout is identical to DeepSeek-OCR), auto_map removed; processor_config.json: processor_class → DeepseekOCRProcessor.
2. mlx.nn.quantize on the BF16 model with a per-module predicate (4-bit experts/MLP, 8-bit attention/embeddings/LM head; vision tower and projector kept float) (script to be published with the eval harness repository).
3. BF16 conversion smoke-tested on a rendered document before any quantization; every published variant evaluated afterward (table above).

Limitations — read this

• R-SWA is not implemented in mlx-vlm. The base model's headline mechanism (Reference Sliding Window Attention, constant KV cache for long-horizon parsing) is replaced by standard full attention in this and every current MLX port (re-verified 2026-07-02). Update: a reference implementation now exists in llama.cpp (open PR ggml-org/llama.cpp#24975), and a measured single-page comparison on identical weights and this same corpus found R-SWA and full attention produced identical recognized text on 21 of 24 pages (see the GGUF release in this series' collection) — supporting single-page full-attention results as a reasonable proxy. The multi-page constant-memory regime remains unexercised, and an R-SWA implementation for MLX remains a planned follow-up.
• Synthetic corpus: clean renders, three fonts, English. Real-world scans (skew, noise, handwriting, other scripts) are not covered by these numbers.
• Quality figures are specific to this corpus and mlx-vlm 0.6.3; rerun the harness for your own document distribution before trusting a lower-bit variant in production.

Attribution

Base model © Baidu, released under MIT — please cite their technical report. This repo contains converted/quantized weights and adds no training.