research-document-archive

Research Document Archive

234,630 declassified U.S. government documents processed through a 13-step ML pipeline. 3.2 million pages OCR'd, 31 million named entities extracted and linked, 288 topic clusters identified.

Live platform: tanglewoodapp.com

Collections

ML Pipeline (13 Steps)

1. Document ingestion and format normalization
2. OCR with Tesseract + post-correction
3. Classification stamp detection (SECRET, CONFIDENTIAL, UNCLASSIFIED, etc.)
4. Redaction detection and boundary mapping
5. Named entity recognition (people, organizations, locations, dates)
6. Entity disambiguation and cross-document linking
7. Relationship extraction
8. Topic modeling (LDA + BERTopic)
9. Timeline event extraction
10. Network graph construction
11. Sentiment and tone analysis
12. Document similarity clustering
13. Index building for search and retrieval

Classification Stamps Detected

Key Statistics

• 31M named entities extracted
• 2.9M entity cross-document links
• 59,830 redactions detected and mapped
• 288 topic clusters identified
• 6 document collections spanning 1860s–2000s

Usage

from datasets import load_dataset

ds = load_dataset("datamatters24/research-document-archive")

# Filter by collection
jfk = ds.filter(lambda x: x["collection"] == "jfk_assassination")

# Search by entity
cia_docs = ds.filter(lambda x: "CIA" in x["entities"])

Data Sources

All documents are public record obtained from:

• National Archives (NARA)
• CIA FOIA Reading Room
• Congress.gov
• Library of Congress

Citation

@misc{rubin2026researcharchive,
  author = {Rubin, Theodore},
  title = {Research Document Archive: ML Pipeline for Declassified U.S. Government Documents},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/datasets/datamatters24/research-document-archive}
}