unlimited-ocr-6bit-mlx

Unlimited-OCR — MLX 6-bit (vision towers bf16)

MLX conversion of baidu/Unlimited-OCR (3.34B MoE, SAM ViT-B + CLIP-L dual vision encoders, DeepSeek-V2 MoE decoder), converted with mlx-vlm 0.6.17 for Apple Silicon.

Quantization: 6-bit affine on the language stack; vision towers (SAM + CLIP) left in bf16. 7.646 bits/weight overall, 3.19 GB on disk.

Why this one

At the time of writing there are 35 MLX conversions of this model on the Hub. Five ship the complete upstream tokenizer, and this is the only one of those that is quantized — the other four are 6.67 GB full precision. On a 16 GB Mac that is the difference between 5.15 GB and 8.64 GB peak memory, and it runs 1.94x faster (measured below).

If size is not a constraint, pcuenq/Unlimited-OCR-bf16-mlx is a clean, properly-indexed full-precision conversion from Hugging Face's own MLX maintainer and is a perfectly good default.

Correction (2026-10-04)

An earlier version of this card claimed that conversions shipping a stripped tokenizer_config.json would load a slow tokenizer, emit raw <0x..> byte markers, and fail to round-trip the grounding markers. That was wrong, and the correction is posted upstream.

On transformers 5.x those conversions behave identically to upstream: fast tokenizer, all five grounding markers single-token, exact round-trip under skip_special_tokens=False. added_tokens_decoder is redundant for a fast tokenizer, because tokenizer.json carries the definitions and is copied intact.

The real difference is narrower: this conversion restores the full upstream tokenizer files, so it also loads under transformers 4.x, where the stripped ones raise AttributeError: 'list' object has no attribute 'keys'. That matters only if you load this repo's tokenizer with 4.x — relevant here because the model's own custom modeling code targets 4.x. It ships no chat_template.jinja, matching upstream, so mlx-vlm's no-template path passes prompts through verbatim.

Known limitation: repetition loops on dense non-English pages

This is the most important thing on this card, and it is a property of the model under mlx-vlm rather than of this conversion.

On a 28-page corpus (synthetic multilingual/dense/degraded pages plus public-domain Chronicling America newspaper scans), with decoding policy matched on both sides, 13 of 28 pages degenerated into repetition loops — 6 on both CUDA and MLX, 4 on CUDA only, 3 on MLX only — including every non-English scanned page (German Fraktur, French, Italian, Serbian/Cyrillic, Yiddish/Hebrew). These occurred under <image>document parsing., the recommended prompt, not the Free OCR. prompt usually blamed.

No available decoding control reliably fixes this (measured 2026-10-05):

Where the n-gram filter works it works decisively (one page went from compression ratio 0.005 to 0.461; another shrank from 10,845 characters of looped output to 470). But five pages still loop with it enabled.

mlx-vlm exposes repetition_penalty, frequency_penalty, presence_penalty and arbitrary logits_processors, but not no_repeat_ngram_size — a soft logit penalty rather than hard n-gram blocking. The penalty is not a substitute: it fixed 2 of 8 looping pages, left 6, and pushed a previously clean page into a loop, with no monotonic relationship between penalty strength and outcome. Setting repetition_penalty here is as likely to hurt as help.

Practical advice: budget for failures on dense non-English documents rather than expecting to configure them away. A cheap detector is the zlib compression ratio of the output text — clean pages sit at 0.25-0.77, looped ones below 0.20.

Verification

Apple M4, 16 GB, macOS 26.6.2; mlx-vlm 0.6.17 / mlx 0.32.2 / transformers 5.16.1. CUDA reference produced with the upstream transformers implementation on an RTX 3060, same rasters, same decoding policy.

Excluding degenerate pages (n=15):

Quantization costs 14-40x less than the port to MLX itself. Across all 28 pages this build produced 690 grounding blocks to bf16's 685 (0.7% difference) at 28.2 s/page vs 54.9 — so the quantization is close to free in quality and nearly halves the runtime.

The scanned CER rests on 5 English pages; 11 of 16 scanned pages were excluded as degenerate per the section above.

Tokenizer checks: 165,938-byte tokenizer_config.json, 830 added_tokens_decoder entries, fast tokenizer, 5/5 grounding markers round-trip under skip_special_tokens=False, zero byte markers across the corpus.

Usage

from mlx_vlm import load, generate

model, processor = load("wittjeff/unlimited-ocr-6bit-mlx", trust_remote_code=True)
out = generate(
    model, processor,
    "<image>document parsing.",   # keep the exact wording — the model is prompt-sensitive
    image=["page.png"],
    max_tokens=8192, temperature=0.0,
)
print(out.text)

Known differences vs the CUDA reference implementation

• mlx-vlm runs full attention in place of the model's reference sliding-window attention (R-SWA).
• No no_repeat_ngram_size / ngram_window filter — see above.
• Gundam (cropped 640 px) dynamic-resolution mode is untested; verification used base mode.

License

MIT, inherited from upstream. Weights (c) Baidu, converted without modification beyond format and quantization.