review-knowledge-base-tool

Review Knowledge Base

A RAG (Retrieval-Augmented Generation) system that scrapes academic paper reviews from OpenReview and builds a searchable vector database. Designed for AI agents to perform Precedent Checks — verifying whether a criticism of a paper aligns with historical review norms.

What's Inside

Supported Conferences

• ICLR 2023–2026
• NeurIPS 2023–2026
• ICML 2023–2025
• ACL ARR 2025 (monthly cycles: Feb, Apr, Jun, Aug, Oct, Dec)

Quick Start

1. Install Dependencies

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Set Credentials (Optional)

export OPENREVIEW_USERNAME="[email protected]"
export OPENREVIEW_PASSWORD="your-password"
export HF_TOKEN="hf_xxxxxxx"

3. Run the Pipeline

# Full pipeline
./run_pipeline.sh

# Or step by step
python 01_scrape_openreview.py
python 02_build_chromadb.py --require-decision --require-meta-review --rebuild

4. Query

# Interactive
python 04_query_rag.py

# Single query
python 04_query_rag.py --query "using TTS synthetic data for training"

# Precedent check
python 04_query_rag.py --query "weak evaluation methodology" --check

# JSON output (for agents)
python 04_query_rag.py --query "..." --json

5. Agent Integration (Python)

from agent_integration import ReviewerRAG

rag = ReviewerRAG()
result = rag.check_criticism(
    paper_abstract="We propose a novel TTS model using synthetic data...",
    criticism="The paper relies on synthetic data which is unreliable."
)
print(result["is_valid_criticism"])  # True/False
print(result["suggestion"])          # How to revise the criticism

Data Schema

Each record contains:

Using with HuggingFace

# Push data
python 03_push_to_hf.py --require-decision

# Pull data on another machine
python 03_push_to_hf.py --pull

License

MIT