emet-occrp-demo

Emet

AI-Powered Investigative Intelligence for Anti-Corruption Journalism

Emet is an autonomous investigation agent that traces corporate ownership, screens sanctions lists, analyzes blockchain flows, and synthesizes findings into auditable, publication-safe reports. It is built on the FollowTheMoney (FtM) data model -- the same schema used by Aleph and OCCRP's investigative data platform.

The name "Emet" (Hebrew: truth) is the word inscribed on the forehead of the Golem -- a guardian animated to protect its community. The first letter, Aleph, is also the name of OCCRP's investigative data platform.

---

What It Does

Give Emet a goal in plain language. It runs a multi-step investigation autonomously:

emet investigate \
  "Trace beneficial ownership of Meridian Holdings through offshore jurisdictions" \
  --llm ollama --max-turns 8

An LLM reasons about what to do next. The agent searches entity databases, screens sanctions and PEP lists, traces corporate ownership chains, monitors global news, investigates blockchain transactions, and synthesizes everything into a structured report -- with PII scrubbed at publication boundaries and a full forensic audit trail.

Investigation Tools

Additional Capabilities

• Federated search across 7+ sources with parallel async fan-out, Jaccard deduplication, token-bucket rate limiting, and response caching
• Evidence chain tracking: every factual claim is bound to source references with a 0-1 confidence score; unsupported claims are flagged before publication
• Congressional STOCK Act disclosure analysis (House Clerk feed + FEC campaign finance)
• CourtListener/RECAP federal court record integration
• SEC EDGAR real-time filing feed (Form 4, SC 13D, 8-K)
• On-chain intelligence: OFAC-designated mixer detection (Tornado Cash, Sinbad), DeFi protocol labeling, exchange identification, wallet clustering via co-spend heuristic
• Interactive graph visualization (self-contained Cytoscape.js HTML, no CDN dependency)
• JSON-LD export with schema.org vocabulary mapping
• FtM bundle export (JSONL/zip) for round-trip Aleph re-import
• Entity resolution via Splink probabilistic linkage with pure-Python fallback
• Temporal pattern detection: burst analysis, coincidence detection across entity timelines

---

Architecture

Goal (natural language)
  |
  v
+----------------------------------------------------------------+
|  Agent Loop (InvestigationAgent)                                |
|  LLM decision -> tool execution -> result processing           |
|  One tool call per turn, budget-capped, with full audit trail   |
|  Heuristic fallback when LLM is unavailable                    |
+------+------------------+-------------------+---------+--------+
       |                  |                   |         |
  +----v-----+    +-------v--------+   +------v-----+  |
  | Safety   |    | Session State  |   | Tool       |  |
  | Harness  |    |                |   | Executor   |  |
  |          |    | FtM entities   |   | (MCP)      |  |
  | PII scrub|    | Findings+conf  |   |            |  |
  | Audit log|    | Leads+priority |   | 9 tools    |  |
  | Cost cap |    | Reasoning trace|   |            |  |
  +----------+    +----------------+   +------------+  |
                                                       |
  +----------------------------------------------------v--------+
  |  Data Layer                                                  |
  |                                                              |
  |  FtM Data Spine       Graph Engine (NetworkX)                |
  |  Aleph Client         7 investigative algorithms             |
  |  Federated Search     Multi-format export                    |
  |  Blockchain (4 chains)                                       |
  |  Document Sources     Export (Markdown, PDF, FtM, Timeline)  |
  +--------------------------------------------------------------+

Three-Tier Design

1. Collection -- Federated data retrieval from public registries, sanctions lists, blockchain explorers, court records, and news feeds. Each source adapter produces FtM-schema entities with provenance metadata.

2. Analysis -- Graph algorithms (PageRank, community detection via Louvain/label propagation, shortest path, broker identification, circular ownership detection, shell company topology scoring, structural anomaly detection), evidence chain confidence scoring, temporal pattern analysis, and entity resolution.

3. Orchestration -- LLM-driven agent loop that decides which tools to call, tracks leads by priority, manages investigation budget, and synthesizes findings into reports. Falls back to heuristic routing when no LLM is available.

Human review is required at the publication boundary. The safety harness operates in two modes: investigate (audit-only -- log everything, block nothing) and publish (enforcing -- PII scrubbed, sensitive data redacted, all outputs sanitized).

LLM Backend

Emet is LLM-agnostic with a tiered routing system:

Three provider backends:

• Ollama (default) -- local models, zero API cost, data never leaves the machine
• Anthropic Claude -- cloud API fallback
• Stub -- canned responses for testing and demo mode

Cascading fallback: if the configured provider is unavailable, Emet degrades gracefully through the chain (Ollama -> Anthropic -> Stub -> heuristic routing). It never crashes due to LLM unavailability.

Memory

Within-investigation: Three-stage Continuum Memory Architecture (CMA):

• Stage 1: Sliding-window dialogue compression with entropy-based filtering and atomic fact extraction (based on SimpleMem, arXiv:2601.02553)
• Stage 2: Significance scoring, temporal tagging, spaced retrieval scheduling
• Stage 3: Hybrid retrieval -- dense vectors (pgvector), lexical (tsvector/BM25), symbolic (significance score) with adaptive fusion

Cross-investigation: HippoRAG knowledge graph (based on arXiv:2502.14802) with CatRAG query-adaptive edge weighting. Entities from different investigations are linked automatically through co-occurrence triples and Personalized PageRank. This is where "Entity A is a donor in Investigation 1" connects to "Entity A is a contractor in Investigation 2."

Graph Capabilities

Built on NetworkX with investigative interpretations layered on standard algorithms:

All results include human-readable explanations suitable for journalists and suggested follow-up actions.

Export formats: GEXF (Gephi), GraphML, D3 JSON, Cytoscape JSON, CSV, interactive HTML (Cytoscape.js).

FtM / Aleph Interoperability

All entities use the FollowTheMoney data model throughout the pipeline. This means:

• Entities produced by Emet can be exported as FtM JSONL bundles and imported directly into Aleph collections
• Aleph collections can be searched and retrieved natively via the search_aleph tool
• Entity schemas (Person, Company, LegalEntity, Ownership, Directorship, Payment, etc.) are consistent with OCCRP's existing data infrastructure
• Federated search results from all sources are normalized to FtM before entering the investigation graph

Security

Ethics Governance

Emet's behavior is governed by VALUES.json, a machine-readable ethics constitution based on the SPJ Code of Ethics, OCCRP editorial standards, and GIJN verification methodology:

Consensus gates require human editorial approval for publication, entity modification, and sensitive operations.

---

Deployment

Docker (recommended)

Three deployment tiers, all fully local -- no data leaves the machine, no cloud APIs called:

# Field -- laptop / Mac Mini (24-32GB RAM)
# Models: Qwen3 14B + Qwen3 8B (~20GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.field.yml up -d

# Server -- office server (64GB+ RAM)
# Models: Qwen3.5 27B + Qwen3 8B (~40GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.server.yml up -d

# Enterprise -- newsroom infrastructure (256GB+ RAM)
# Models: Qwen3 235B MoE + Qwen3.5 27B + Qwen3 8B (~170GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.enterprise.yml up -d

On first run, the ollama-setup container pulls models automatically. Monitor progress:

docker compose logs -f ollama-setup

Verify the stack is running:

docker compose ps
# engine, db, redis, mcp, spiderfoot should be running

curl http://localhost:8000/health
# {"status": "ok"}

Local Development (pip)

Requires Python 3.11+.

pip install -e ".[dev]"
cp .env.example .env
# Edit .env with API keys (all optional -- demo mode works without any)

# Demo mode -- bundled scenario, no keys needed
emet investigate "Meridian Holdings offshore network" --llm stub --demo

# With local Ollama
emet investigate "Trace ownership of Acme Holdings" --llm ollama

# With Anthropic Claude
export ANTHROPIC_API_KEY=sk-ant-...
emet investigate "Trace ownership of Acme Holdings" --llm anthropic

Interfaces

API docs are auto-generated at http://localhost:8000/docs when the HTTP server is running.

Demo Script

# After Docker stack is running:
./demo_occrp.sh

Runs three investigations demonstrating ownership tracing, sanctions screening, and financial flow analysis. Results saved to ./investigations/.

---

Configuration

All data source API keys are optional. Emet degrades gracefully when sources are unavailable, and demo mode works with zero configuration.

See .env.example for the full list of configurable environment variables, grouped by category:

• Data sources -- OpenSanctions, OpenCorporates, UK Companies House, ICIJ, GLEIF, SEC EDGAR, FEC, CourtListener
• Blockchain -- Etherscan (ETH), Solana (public RPC, no key)
• News/OSINT -- GDELT (no key), SpiderFoot (self-hosted)
• Documents -- Datashare (ICIJ), DocumentCloud
• LLM -- Anthropic Claude, Ollama (local)
• Infrastructure -- PostgreSQL, Redis, Aleph

---

Testing

# Full unit + integration suite (~1,650 tests, ~3 minutes)
python -m pytest tests/ -q --ignore=tests/live

# Live integration tests (requires API keys)
python -m pytest -m live tests/live/ -v

# Key test modules
python -m pytest tests/test_agent.py               # Agent loop + session
python -m pytest tests/test_mcp_server.py           # MCP server + tool dispatch
python -m pytest tests/test_graph.py                # Graph analytics
python -m pytest tests/test_federation.py           # Federated search
python -m pytest tests/test_e2e_pipeline.py         # End-to-end pipeline
python -m pytest tests/test_security_pii.py         # PII detection + scrubbing
python -m pytest tests/test_export.py               # Export pipeline
python -m pytest tests/test_mock_fidelity.py        # Tool + demo mode fidelity

No external services required for the unit suite. All tests use stubs, mocks, and synthetic datasets.

---

Project Structure

emet/
  agent/                    # Autonomous investigation engine
    loop.py                 # InvestigationAgent: LLM decisions, tool execution
    session.py              # Investigation state (entities, findings, leads)
    safety_harness.py       # Two-mode safety: audit-only vs enforcing
    audit.py                # Forensic audit archive (gzip JSONL, SHA-256)
    persistence.py          # Session save/load for investigation resume
  mcp/                      # Model Context Protocol server
    server.py               # MCP server (stdio / SSE / HTTP transport)
    tools.py                # 9 MCP tools + EmetToolExecutor
    resources.py            # MCP resource providers
  api/                      # HTTP API (FastAPI)
    app.py                  # App factory
    websocket.py            # WebSocket streaming
    routes/                 # REST endpoints
  cognition/                # LLM abstraction
    llm_base.py             # LLMClient ABC
    llm_anthropic.py        # Anthropic Claude
    llm_ollama.py           # Ollama (local)
    llm_stub.py             # Canned responses (testing)
    llm_factory.py          # Provider factory + cascading fallback
    model_router.py         # Tier-to-model mapping
  ftm/                      # FollowTheMoney integration
    data_spine.py           # FtM entity factory
    aleph_client.py         # Async Aleph REST client
    external/               # Federated data sources
      federation.py         # Parallel async fan-out (7+ sources)
      adapters.py           # OpenSanctions, OpenCorporates, ICIJ, GLEIF
      companies_house.py    # UK Companies House
      edgar.py              # SEC EDGAR + real-time Atom feed
      congress.py           # Congressional STOCK Act disclosures
      fec.py                # FEC campaign finance
      courtlistener.py      # CourtListener / RECAP
      blockchain.py         # ETH, BTC, Tron, Solana
      crypto_intel.py       # Mixer detection, DeFi labels, risk scoring
      converters.py         # Source -> FtM converters
      rate_limit.py         # Token bucket, monthly counter, cache
  graph/                    # Network analysis
    algorithms.py           # 8 investigative algorithms + UBO tracer
    engine.py               # GraphEngine orchestrator
    ftm_loader.py           # FtM -> NetworkX conversion
    exporters.py            # GEXF, GraphML, CSV, D3, Cytoscape, JSON-LD
    visualizer.py           # Interactive Cytoscape.js HTML
  export/                   # Investigation output
    markdown.py             # Markdown reports
    pdf.py                  # PDF reports (reportlab)
    ftm_bundle.py           # FtM JSONL/zip for Aleph re-import
    evidence.py             # Evidence chain + confidence scoring
    timeline.py             # Temporal analysis + interactive HTML timeline
  security/                 # Security layer
    pii.py                  # PII detection + redaction
    target_policy.py        # Public figures only enforcement
    intent_capsule.py       # HMAC-SHA256 signed action mandates
    shield.py               # Egress allowlist, budget enforcement
    monitor.py              # Post-execution anomaly detection
    sandbox.py              # Execution sandbox
  memory/                   # Continuum Memory Architecture
    cma_stage1.py           # Semantic compression
    cma_stage2.py           # Significance scoring + spaced retrieval
    cma_stage3.py           # Hybrid retrieval (dense + lexical + symbolic)
  governance/               # Ethics enforcement
    consensus.py            # Consensus gates for publication actions
  config/                   # Settings + VALUES.json loader
  monitoring/               # Change detection + sanctions alerts
  workflows/                # Predefined investigation templates
  cli.py                    # CLI entry point
mnemosyne/                  # Cross-investigation memory modules
  hipporag-catrag-kg/       # Knowledge graph (HippoRAG + CatRAG)
  kintsugi-cma/             # Reference CMA implementation
  sira-enrichment/          # Semantic Indexed Retrieval
  tgs-rag-bridge/           # Memory-to-graph bridge
  tgs-verification/         # Memory verification
skills/                     # Investigation methodology docs (SKILL.md)
tests/                      # ~1,650 unit tests + 44 live integration tests
VALUES.json                 # Machine-readable ethics constitution

---

Data Sources

---

License

Hippocratic License 3.0 with AI Welfare module. See LICENSE.md.

Enabled modules: CL (labor rights), ECO (environmental), MEDIA (disinformation), MIL (military), SV (surveillance), XUAR (human rights).

Usage rights by organization type:

• Investigative journalists, newsrooms, press freedom organizations, anti-corruption NGOs, academic journalism programs -- free, no restrictions beyond the license terms.
• Commercial use (compliance, KYC/AML, due diligence, corporate intelligence) -- commercial license required. See LICENSE-COMMERCIAL-ADDENDUM.md.
• Prohibited under all circumstances -- surveillance of journalists or sources, press suppression, targeting whistleblowers, mass surveillance, circumventing press freedom protections.

---

Provenance

Built by Liberation Labs.

Core infrastructure derived from Project Kintsugi (self-repairing agentic harness). The governance, security, memory, and plugin layers are domain-agnostic by design. The investigative layers (agent loop, MCP tools, FtM spine, federated data sources, graph algorithms) are Emet-specific.

Key dependencies: FollowTheMoney (MIT), Aleph (MIT), FastAPI (MIT), NetworkX (BSD), httpx (BSD-3), Pydantic (MIT), SQLAlchemy (MIT), reportlab (BSD).