MonkeyOCR-Apple-Silicon

🚀 MonkeyOCR-MLX: Apple Silicon Optimized OCR

A high-performance OCR application optimized for Apple Silicon with MLX-VLM acceleration, featuring advanced document layout analysis and intelligent text extraction.

🔥 Key Features

• ⚡ MLX-VLM Optimization: Native Apple Silicon acceleration using MLX framework
• 🚀 3x Faster Processing: Compared to standard PyTorch on M-series chips
• 🧠 Advanced AI: Powered by Qwen2.5-VL model with specialized layout analysis
• 📄 Multi-format Support: PDF, PNG, JPG, JPEG with intelligent structure detection
• 🌐 Modern Web Interface: Beautiful Gradio interface for easy document processing
• 🔄 Batch Processing: Efficient handling of multiple documents
• 🎯 High Accuracy: Specialized for complex financial documents and tables
• 🔒 100% Private: All processing happens locally on your Mac

📊 Performance Benchmarks

Test: Complex Financial Document (Tax Form)

• MLX-VLM: ~15-18 seconds ⚡
• Standard PyTorch: ~25-30 seconds
• CPU Only: ~60-90 seconds

MacBook M4 Pro Performance:

• Model loading: ~1.7s
• Text extraction: ~15s
• Table structure: ~18s
• Memory usage: ~13GB peak

🛠 Installation

Prerequisites

• macOS with Apple Silicon (M1/M2/M3/M4)
• Python 3.11+
• 16GB+ RAM (32GB+ recommended for large documents)

Quick Setup

1. Clone the repository:

   git clone https://huggingface.co/Jimmi42/MonkeyOCR-Apple-Silicon
   cd MonkeyOCR-Apple-Silicon
   

2. Run the automated setup script:

   chmod +x setup.sh
   ./setup.sh
   

   This script will automatically:

   • Download MonkeyOCR from the official GitHub repository
   • Apply MLX-VLM optimization patches for Apple Silicon
   • Enable smart backend auto-selection (MLX/LMDeploy/transformers)
   • Install UV package manager if needed
   • Set up virtual environment with Python 3.11
   • Install all dependencies including MLX-VLM
   • Download required model weights
   • Configure optimal backend for your hardware

3. Alternative manual installation:

   # Install UV if not already installed
   curl -LsSf https://astral.sh/uv/install.sh | sh
   
   # Download MonkeyOCR
   git clone https://github.com/Yuliang-Liu/MonkeyOCR.git MonkeyOCR
   
   # Install dependencies (includes mlx-vlm)
   uv sync
   
   # Download models
   cd MonkeyOCR && python tools/download_model.py && cd ..
   

🏃‍♂️ Usage

Web Interface (Recommended)

# Activate virtual environment
source .venv/bin/activate  # or `uv shell`

# Start the web app
python app.py

Access the interface at http://localhost:7861

Command Line

python main.py path/to/document.pdf

⚙️ Configuration

Smart Backend Selection (Default)

The app automatically detects your hardware and selects the optimal backend:

# model_configs_mps.yaml
device: mps
chat_config:
  backend: auto  # Smart auto-selection
  batch_size: 1
  max_new_tokens: 256
  temperature: 0.0

Auto-Selection Logic:

• 🍎 Apple Silicon (MPS) → MLX-VLM (3x faster)
• 🖥️ CUDA GPU → LMDeploy (optimized for NVIDIA)
• 💻 CPU/Fallback → Transformers (universal compatibility)

Performance Backends

🧠 Model Architecture

Core Components

• Layout Detection: DocLayout-YOLO for document structure analysis
• Vision-Language Model: Qwen2.5-VL with MLX optimization
• Layout Reading: LayoutReader for reading order optimization
• MLX Framework: Native Apple Silicon acceleration

Apple Silicon Optimizations

• Metal Performance Shaders: Direct GPU acceleration
• Unified Memory: Optimized memory access patterns
• Neural Engine: Utilizes Apple's dedicated AI hardware
• Float16 Precision: Optimal speed/accuracy balance

🎯 Perfect For

Document Types:

• 📊 Financial Documents: Tax forms, invoices, statements
• 📋 Legal Documents: Contracts, forms, certificates
• 📄 Academic Papers: Research papers, articles
• 🏢 Business Documents: Reports, presentations, spreadsheets

Advanced Features:

• ✅ Complex table extraction with highlighted cells
• ✅ Multi-column layouts and mixed content
• ✅ Mathematical formulas and equations
• ✅ Structured data output (Markdown, JSON)
• ✅ Batch processing for multiple files

🚨 Troubleshooting

MLX-VLM Issues

# Test MLX-VLM availability
python -c "import mlx_vlm; print('✅ MLX-VLM available')"

# Check if auto backend selection is working
python -c "
from MonkeyOCR.magic_pdf.model.custom_model import MonkeyOCR
model = MonkeyOCR('model_configs_mps.yaml')
print(f'Selected backend: {type(model.chat_model).__name__}')
"

Performance Issues

# Check MPS availability
python -c "import torch; print(f'MPS available: {torch.backends.mps.is_available()}')"

# Monitor memory usage during processing
top -pid $(pgrep -f "python app.py")

Common Solutions

1. Patches Not Applied:

   • Re-run ./setup.sh to reapply patches
   • Check that MonkeyOCR directory exists and has our modifications
   • Verify MonkeyChat_MLX class exists in MonkeyOCR/magic_pdf/model/custom_model.py

2. Wrong Backend Selected:

   • Check hardware detection with python -c "import torch; print(torch.backends.mps.is_available())"
   • Verify MLX-VLM is installed: pip install mlx-vlm
   • Use backend: mlx in config to force MLX backend

3. Slow Performance:

   • Ensure auto-selection chose MLX backend on Apple Silicon
   • Check Activity Monitor for MPS GPU usage
   • Verify backend: auto in model_configs_mps.yaml

4. Memory Issues:

   • Reduce image resolution before processing
   • Close other memory-intensive applications
   • Reduce batch_size to 1 in config

5. Port Already in Use:

   GRADIO_SERVER_PORT=7862 python app.py
   

📁 Project Structure

MonkeyOCR-MLX/
├── 🌐 app.py                    # Gradio web interface
├── 🖥️ main.py                   # CLI interface  
├── ⚙️ model_configs_mps.yaml    # MLX-optimized config
├── 📦 requirements.txt          # Dependencies (includes mlx-vlm)
├── 🛠️ torch_patch.py           # Compatibility patches
├── 🧠 MonkeyOCR/               # Core AI models
│   └── 🎯 magic_pdf/           # Processing engine
├── 📄 .gitignore               # Git ignore rules
└── 📚 README.md                # This file

🔥 What's New in MLX Version

• ✨ Smart Patching System: Automatically applies MLX-VLM optimizations to official MonkeyOCR
• 🧠 Intelligent Backend Selection: Auto-detects hardware and selects optimal backend
• 🚀 3x Faster Processing: MLX-VLM acceleration on Apple Silicon
• 💾 Better Memory Efficiency: Optimized for unified memory architecture
• 🎯 Improved Accuracy: Enhanced table and structure detection
• 🔧 Zero Configuration: Works out-of-the-box with smart defaults
• 📊 Performance Monitoring: Built-in timing and metrics
• 🛠️ Latest Fix (June 2025): Resolved MLX-VLM prompt formatting for optimal OCR output
• 🔄 Always Up-to-Date: Uses official MonkeyOCR repository with our patches applied

🔬 Technical Implementation

Smart Patching System

• Dynamic Code Injection: Automatically adds MLX-VLM class to official MonkeyOCR
• Backend Selection Logic: Patches smart hardware detection into initialization
• Zero Maintenance: Always uses latest official MonkeyOCR with our optimizations
• Seamless Integration: Patches are applied transparently during setup

MLX-VLM Backend (MonkeyChat_MLX)

• Direct MLX framework integration
• Optimized for Apple's Metal Performance Shaders
• Native unified memory management
• Specialized prompt processing for OCR tasks
• Fixed prompt formatting for optimal output quality

Intelligent Fallback System

• Hardware Detection: MPS → MLX, CUDA → LMDeploy, CPU → Transformers
• Graceful Degradation: Falls back to compatible backends if preferred unavailable
• Cross-Platform: Maintains compatibility across all systems
• Error Recovery: Automatic fallback on initialization failures

🤝 Contributing

We welcome contributions! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Apple MLX Team: For the incredible MLX framework
• MonkeyOCR Team: For the foundational OCR model
• Qwen Team: For the excellent Qwen2.5-VL model
• Gradio Team: For the beautiful web interface
• MLX-VLM Contributors: For the MLX vision-language integration

📞 Support

• 🐛 Bug Reports: Create an issue
• 💬 Discussions: Hugging Face Discussions
• 📖 Documentation: Check the troubleshooting section above
• ⭐ Star the repository if you find it useful!

---

🚀 Supercharged for Apple Silicon • Made with ❤️ for the MLX Community

Experience the future of OCR with native Apple Silicon optimization