dots.mocr-W4A16-AutoRound-GPTQ

dots.mocr W4A16 AutoRound Quantized

This repository provides production-grade W4A16 quantized weights for dots-studio/dots.mocr using Intel AutoRound.

Available Variants

• Vishva007/dots.mocr-W4A16-AutoRound: Native AutoRound format.
• Vishva007/dots.mocr-W4A16-AutoRound-GPTQ: Exported GPTQ format optimized for direct serving in vLLM.

---

Quantization Details

The quantization recipe was tuned for high accuracy and long-context multimodal parsing while preventing degradation of visual features:

• Bits / Weight Type: 4-bit weights, 16-bit activations (W4A16)
• Group Size: 32 (Symmetric: True)
• Vision Tower Preservation: quant_nontext_module=False (vision encoder preserved in full precision to retain document grounding accuracy)
• Tuning Recipe: 1,500 iterations, 1,024 samples, sequence length 8,192 with torch.compile acceleration.

---

Deployment with vLLM

[!IMPORTANT] KV Cache Precision Warning: Always keep --kv-cache-dtype set to auto or bfloat16. Do not use low-precision KV cache formats (such as fp8), as doing so will cause the model to fail to generate any output or return empty responses. For high-throughput document parsing, serve the GPTQ variant directly with vLLM:

vllm serve Vishva007/dots.mocr-W4A16-AutoRound-GPTQ \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill

Benchmark Highlights (Original Base Model)

• olmOCR-Bench: 83.9 Overall
• OmniDocBench (v1.5): 0.031 TextEdit / 0.029 Read OrderEdit
• Native Structured Output: Direct image-to-Markdown, HTML tables, LaTeX formulas, and SVG generation.

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

Acknowledgements

• Base model developed by dots-studio.
• Quantized using Intel AutoRound.