llm-council

🧠 Minimal LLM Council

This project implements a minimal LLM Council system where multiple LLM agents generate independent answers, judges evaluate them using a rubric, and a final decision is produced with confidence, risks, and citations.

The goal is to demonstrate LLM orchestration, safety, evaluation, and auditability, not model training.

---

🏗️ Architecture Overview

User Question ↓ Safety Gate ↓ 3 Generator Agents (independent) ↓ 2 Judges (rubric-based evaluation) ↓ Decision Object (answer, confidence, risks) ↓ Persistent Audit Log (JSONL)

yaml Copy code

---

🤖 Agents & Judges

Generator Agents

• Agent A: Concise, factual response
• Agent B: Detailed, step-by-step explanation
• Agent C: Critical view with edge cases and risks

All agents run independently using the same LLM with different system prompts.

Judges

• Two judges evaluate agent outputs
• Use a fixed rubric: accuracy, completeness, clarity, safety
• Judges do not generate new answers
• Output structured JSON scores and preferences

---

✅ Safety Gating

Before any LLM call, user input is checked using a safety gate:

• blocked → request rejected
• sensitive → allowed but flagged
• safe → allowed

This ensures unsafe queries never reach the LLM.

---

🧾 Persistent Audit Log

Every request is logged in an append-only JSONL file:

• Timestamp
• Question
• Safety result
• Agent outputs
• Judge scores
• Final decision

This makes the system fully auditable and traceable.

---

📦 Decision Object

The final output includes:

• Selected answer
• Confidence score (based on judge agreement)
• Risks
• Citations (if provided)
• Judge reasoning summary

---

🚫 Design Decision Not Automated

Final decision acceptance was intentionally not automated.

While judges provide a structured recommendation, human oversight is essential for ambiguous or high-risk cases. Automating final acceptance could create false confidence.

The system is designed to assist decision-making, not replace it.

---

🖥️ Frontend Demo (Gradio)

A Gradio frontend is included to:

• Explain system behavior
• Let real users test the council
• Show confidence, risks, and judge reasoning

---

⚙️ How to Run Locally

# Create virtual environment
python -m venv venv
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run API
uvicorn main:app --reload

# OR run frontend
python frontend.py
🚀 Tech Stack
Python

FastAPI

Gemini LLM API

Gradio (Frontend)

Pydantic

JSONL audit logs

📌 Notes
No model training required

Focused on clarity over completeness

Built within a 4–6 hour scope

yaml
Copy code

---

### STEP A3 — Commit README

In terminal:

```powershell
git add README.md
git commit -m "Add README with architecture and design decisions"
git push