kdd1862

MinerU-HTML

MinerU-HTML is an intelligent HTML main content extraction tool based on Large Language Models (LLM). It can accurately identify and extract main content from complex web page HTML, automatically filtering out auxiliary elements such as navigation bars, advertisements, and metadata.

✨ Features

• 🎯 LLM-Powered Extraction: Uses state-of-the-art language models to intelligently identify main content
• ⚡ Compact Format: Now use compact format for the model instead of the original JSON format for faster inference speed
• 🔄 Complete Processing Pipeline: Includes HTML simplification, prompt construction, LLM inference, result parsing, main content extraction, and other complete processes
• 🚀 Multiple Inference Backend Support: Supports three inference backends: VLLM, Transformers, and OpenAI API
• 🛡️ Fault Tolerance: Supports fallback mechanism (trafilatura or bypass) to ensure content extraction even when processing fails
• 🔌 Modular Design: Clear architecture design, easy to extend and customize
• 🧪 Complete Testing: Includes comprehensive unit tests and integration tests

📋 Directory Structure

MinerU-HTML/
├── mineru_html/             # Core code package
│   ├── __init__.py          # Package initialization, exports main API
│   ├── api.py               # Main API interfaces (MinerUHTMLGeneric, MinerUHTMLConfig)
│   ├── base.py              # Base data class definitions
│   ├── constants.py         # Constant definitions
│   ├── exceptions.py        # Exception class definitions
│   ├── utils.py             # Utility functions
│   ├── implementations/     # Concrete implementation classes
│   │   ├── __init__.py      # implementations subpackage initialization file
│   │   ├── vllm_api.py      # VLLM API backend implementation (MinerUHTML)
│   │   ├── openai_api.py    # OpenAI API backend implementation (MinerUHTML_OpenAI)
│   │   └── transformers_api.py  # Transformers API backend implementation (MinerUHTML_Transformers)
│   ├── inference/           # Inference backend abstraction layer
│   │   ├── __init__.py      # inference subpackage initialization file
│   │   ├── base_backend.py  # Inference backend base class (InferenceBackend)
│   │   ├── factory.py       # Backend factory functions
│   │   ├── vllm_backend.py  # VLLM backend implementation (VLLMInferenceBackend)
│   │   ├── transformers_backend.py  # Transformers backend implementation (TransformersInferenceBackend)
│   │   └── openai_backend.py  # OpenAI API backend implementation (OpenaiAPIInferenceBackend)
│   └── process/             # Processing pipeline module
│       ├── __init__.py      # process subpackage initialization file
│       ├── build_prompt.py  # Prompt construction
│       ├── simplify_html.py # HTML simplification
│       ├── parse_result.py  # Result parsing
│       ├── map_to_main.py   # Main content extraction
│       └── html_utils.py    # HTML utility functions
├── tests/                   # Test code
├── benchmark/               # Benchmark test data
├── eval_baselines/          # Benchmark evaluation code
├── requirements/            # Dependency package lists (by module)
│   ├── core.txt            # Core dependencies
│   ├── openai.txt          # OpenAI related dependencies
│   └── vllm.txt            # VLLM related dependencies
├── README.md                # Documentation
├── run_eval.sh              # Benchmark test execution script
├── baselines.txt            # Benchmark dependency package list
├── eval_baselines.py        # Benchmark evaluation entry code
└── setup.py                # Installation configuration (supports optional dependencies)

🚀 Quick Start

Installation

Basic Installation (Core Dependencies Only)

pip install .
# Or development mode
pip install -e .

Install Specific Backend Dependencies

Depending on your use case, you can choose to install different backend dependencies:

# Install OpenAI backend dependencies
pip install .[openai]

# Install VLLM backend dependencies
pip install .[vllm]

# Install all dependencies
pip install .[all]

Basic Usage

Using VLLM Backend

from mineru_html import MinerUHTML, MinerUHTMLConfig

# Create configuration
config = MinerUHTMLConfig(
    use_fall_back='trafilatura',  # or 'bypass'
    prompt_version='compact',     # recommend using compact
    early_load=True
)

# Initialize MinerUHTML (using local model)
extractor = MinerUHTML(
    model_path='path/to/your/model',   # The model used must correspond to the prompt_version in MinerUHTMLConfig
    config=config
)

# Process single HTML
html_content = '<html>...</html>'
result = extractor.process(html_content)
print(result[0].main_html)

# Process multiple HTML
html_list = ['<html>...</html>', '<html>...</html>']
results = extractor.process(html_list)
for result in results:
    print(result.main_html)
    print(result.case_id)

Using OpenAI API Backend

from mineru_html import MinerUHTML_OpenAI, MinerUHTMLConfig

# Create configuration
config = MinerUHTMLConfig(
    use_fall_back='trafilatura',
    prompt_version='v2',          # Recommend using v2
    early_load=True
)

# Initialize MinerUHTML_OpenAI
extractor = MinerUHTML_OpenAI(
    base_url='https://api.openai.com/v1',
    sk='your-api-key',
    model='gpt-5',
    config=config,
    retry_times=3
)

# Process HTML
html_content = '<html>...</html>'
result = extractor.process(html_content)
print(result[0].main_html)

Using Transformers Backend

from mineru_html import MinerUHTML_Transformers, MinerUHTMLConfig

# Create configuration
config = MinerUHTMLConfig(
    use_fall_back='trafilatura',
    prompt_version='v2',
    early_load=True
)

# Initialize MinerUHTML_Transformers
extractor = MinerUHTML_Transformers(
    model_path='path/to/your/model',   # The model used must correspond to the prompt_version in MinerUHTMLConfig
    config=config,
    model_init_kwargs={
        'device_map': 'auto',
        'dtype': 'auto',
    },
    model_gen_kwargs={
        'max_new_tokens': 8192,
    }
)

# Process HTML
html_content = '<html>...</html>'
result = extractor.process(html_content)
print(result[0].main_html)

📖 Core Concepts

Processing Pipeline

The MinerU-HTML processing pipeline includes the following steps:

1. HTML Simplification (simplify_html): Simplifies raw HTML into a structured format, assigning a unique _item_id attribute to each element
2. Prompt Construction (build_prompt): Constructs LLM prompts based on simplified HTML to guide the model in content classification
3. LLM Inference (inference): Uses LLM to classify each element, marking them as "main" (main content) or "other" (auxiliary content)
4. Result Parsing (parse_result): Parses the classification results returned by LLM
5. Main Content Extraction (extract_main_html): Extracts main content from original HTML based on classification results
6. Fallback Processing: If the above process fails, uses fallback mechanism (trafilatura or bypass) for content extraction

Configuration Options

MinerUHTMLConfig supports the following configurations:

• use_fall_back: Fallback type, optional 'trafilatura' or 'bypass'
• early_load: Whether to load the model early (default True)
• prompt_version: Prompt version, optional 'v2', 'compact'. The MinerUHTML interface and MinerUHTML_Transformers interface can use 'compact', the MinerUHTML_OpenAI interface can use 'v2'

Usage instructions for different prompt_version:

• 'compact': Used for local model inference, it returns more concise results (only keeping the key and value in the JSON dictionary). It is recommended to use the 'compact' model for faster inference speed.
• 'v2': Used for OpenAI API inference, it is the result after prompt optimization.

🔧 Advanced Usage

Using Factory Functions to Create Backends

You can also directly use factory functions to create backends and then pass them to MinerUHTMLGeneric:

from mineru_html import MinerUHTMLGeneric, MinerUHTMLConfig
from mineru_html.inference.factory import create_vllm_backend, create_openai_backend, create_transformers_backend

# Create VLLM backend using factory function
llm = create_vllm_backend(
    model_path='path/to/model',
    response_format='compact',
    max_context_window=32 * 1024,
    model_init_kwargs={'tensor_parallel_size': 1}
)

# Create Transformers backend using factory function
llm = create_transformers_backend(
    model_path='path/to/model',
    max_context_window=32 * 1024,
    model_init_kwargs={
        'device_map': 'auto',
        'dtype': 'auto',
    },
    model_gen_kwargs={
        'max_new_tokens': 8192,
    }
)

# Create OpenAI backend using factory function
llm = create_openai_backend(
    base_url='https://api.openai.com/v1',
    sk='your-api-key',
    model='gpt-5',
    max_context_window=128 * 1000
)

# Use the created backend
config = MinerUHTMLConfig()
extractor = MinerUHTMLGeneric(llm=llm, config=config)

Error Handling

from mineru_html.exceptions import MinerUHTMLError

try:
    result = extractor.process(html_content)
except MinerUHTMLError as e:
    print(f"Processing failed: {e}")
    print(f"Case ID: {e.case_id}")

Backend Selection Recommendations

• VLLM Backend (MinerUHTML): Suitable for local deployment, requires GPU, best performance
• Transformers Backend (MinerUHTML_Transformers): Suitable for local deployment, supports CPU/GPU, high flexibility
• OpenAI API Backend (MinerUHTML_OpenAI): Suitable for cloud services, no local model required, simple to use

🧪 Testing

Run unit tests and integration tests:

pytest tests/