NuExtract3-W4A16-AutoRound-LLM-Compressor

NuExtract3 (W4A16 Quantized via AutoRound)

W4A16 (4-bit integer weights, 16-bit activations) quantized builds of numind/NuExtract3 created using Intel's AutoRound algorithm.

---

⚡ Quantization Details

• Algorithm: Intel AutoRound (v0.14.2)
• Precision / Scheme: W4A16 (Group Size: 32, Symmetric: True)
• Calibration: 512 samples, sequence length 4096, 1000 tuning iterations
• Vision Tower (quant_nontext_module): Kept in BF16 to preserve visual document parsing and OCR accuracy
• Special Modules: Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

---

📦 Available Repositories

• AutoRound: Vishva007/NuExtract3-W4A16-AutoRound
• AutoGPTQ: Vishva007/NuExtract3-W4A16-AutoRound-GPTQ
• LLM-Compressor (vLLM native): Vishva007/NuExtract3-W4A16-AutoRound-LLM-Compressor

---

🚀 vLLM Serving & Inference

1. Launch Server

vllm serve Vishva007/NuExtract3-W4A16-AutoRound-LLM-Compressor \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image": 10, "video": 0}' \
  --chat-template-content-format openai \
  --generation-config vllm \
  --max-model-len 16384 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90

2. Structured Extraction & OCR (Python Client)

import base64
import json
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

data_url = f"data:image/png;base64,{encode_image('document.png')}"

# Structured JSON Extraction
template = {
    "invoice_number": "verbatim-string",
    "date": "date-time",
    "total": "number",
    "line_items": [{"description": "verbatim-string", "amount": "number"}]
}

response = client.chat.completions.create(
    model="Vishva007/NuExtract3-W4A16-AutoRound-LLM-Compressor",
    temperature=0.2,
    messages=[
        {"role": "user", "content": [{"type": "image_url", "image_url": {"url": data_url}}]}
    ],
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

For Markdown OCR: Replace template in chat_template_kwargs with "mode": "markdown".

---

📊 VRAM Benefits

• Base Model (BF16): ~9–11 GB VRAM
• Quantized (W4A16 G32): ~2.8–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs)

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

📚 Acknowledgments

• Base Model: NuMind
• Quantization Engine: Intel AutoRound