faq-policy-qa-bot

FAQ / Policy Q&A Bot

A conversational agent that answers questions grounded in a user-supplied document — company FAQs, HR policies, or product documentation. Rather than relying on a general-purpose chatbot that may hallucinate, the system uses retrieval-augmented generation (RAG) so every answer is traceable to the source text.

Group 3 — MSAI 631, Human-Computer Interaction, University of the Cumberlands Akshay Srinivasan · Manish Basnet · Rabilal Kharel · Satwika Thota · Suyash Sharma

---

Architecture

flowchart TD
    subgraph INGEST["src/ingest.py — Satwika"]
        A[Source document<br/>PDF or TXT] --> B[Extract text]
        B --> C[Clean + normalise]
        C --> D[Split into chunks<br/>CHUNK_SIZE / CHUNK_OVERLAP]
    end

    subgraph RETRIEVE["src/retrieval.py — Suyash"]
        D -->|"list of Chunk"| E[Embed chunks<br/>all-MiniLM-L6-v2]
        E --> F[(Vector index<br/>NumPy or FAISS)]
        G[User question] --> H[Embed query]
        H --> I{Best score ><br/>THRESHOLD?}
        F --> I
        I -->|yes| J[Top-K chunks]
    end

    subgraph GENERATE["src/generation.py — Akshay"]
        J -->|"list of Chunk"| K[Build prompt<br/>context + question]
        K --> L[flan-t5-base]
        L --> M[Answer + citations]
    end

    subgraph UI["app.py — Rabilal"]
        G
        M -->|Answer| N[Display answer]
        M --> O[Display source chunks]
    end

    I -->|no| P[Fallback:<br/>cannot answer<br/>from this document]
    P --> N

    subgraph EVAL["eval/ — Manish"]
        M -.->|scored against| Q[questions.yaml<br/>ground truth]
    end

    style INGEST fill:#e8f0fe,stroke:#4285f4
    style RETRIEVE fill:#e6f4ea,stroke:#34a853
    style GENERATE fill:#fef7e0,stroke:#fbbc04
    style UI fill:#fce8e6,stroke:#ea4335
    style EVAL fill:#f3e8fd,stroke:#a142f4

If the diagram above does not render, here is the same flow as text:

  Document ──► ingest.py ──► [Chunk] ──► retrieval.py ──► [Chunk] ──► generation.py ──► Answer ──► app.py
   (Satwika)                            (Suyash)                     (Akshay)                    (Rabilal)
                                             │
                                             └── below threshold? ──► fallback message
                                                                              │
                                                            eval/ (Manish) scores the Answer

---

Data contract

All five modules communicate through two dataclasses in src/types.py. This file is frozen — changes need group agreement, not a solo commit.

---

Who owns what

Each person owns exactly one file. Do not commit to a file you do not own — if you need something changed there, ask its owner.

Every module ships with a working stub that returns fake-but-valid data. That means the full pipeline runs today, and nobody is blocked waiting for an upstream component. Replace your stub; keep your function signature.

---

Getting started

git clone https://huggingface.co/uc-summer-2026-group-project/faq-policy-qa-bot
cd faq-policy-qa-bot

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

python -m pytest tests/ -v      # smoke test: passes on stubs
python app.py                   # launch the UI locally

Working without stepping on each other

git checkout -b feature/<your-component>     # e.g. feature/retrieval
# ... work only in your own file ...
git add src/your_file.py
git commit -m "Implement <component>"
git push origin feature/<your-component>

Then open a pull request on the Hugging Face repo and merge into main. Branching is not bureaucracy here — the five components are on a shared critical path, and PRs give us a documented process to write up in the results paper.

---

Models

All models are pretrained and used as-is. No training or fine-tuning.

Per the assignment brief, the project deliberately avoids OpenAI and any paid API tokens.

---

Knowledge base

Texas Driver Handbook (Texas Department of Public Safety) — public domain as a US state government work, so it can be redistributed in this repo. It was chosen because it is long, poorly indexed, and full of specific factual rules with objectively verifiable answers, which makes accuracy measurable rather than a matter of opinion.

Included at data/Texas_DrivingLicense_handbook.pdf (DL-7, revised January 2026).

Measured properties, which drive the design decisions below:

Two consequences worth knowing before you start:

1. NumPy cosine similarity is sufficient. At ~600 chunks the index is under a megabyte, so FAISS buys us nothing. It is not in requirements.txt.
2. Every page carries a running header (e.g. "Chapter 5: Signals, Signs, and Markers Texas Driver Handbook", repeated 13 times). This must be stripped during ingestion — left in, it is embedded into all ~600 chunks and degrades every similarity score.

Pages 47 and 76 are sign diagrams with almost no extractable text, so questions about sign shapes and colours cannot be answered. This is a known limitation, not a bug.

---

MVP scope

In scope:

• Single-turn chat (no multi-session memory)
• One document at a time
• Answers that quote the retrieved source chunk so the user can verify
• A fallback response when the question is out of scope

Out of scope (stretch goals, not requirements): multi-document knowledge bases, conversation memory, user accounts.

Known limitations

• Answer quality is bounded by a small, freely runnable model rather than a large commercial one.
• Retrieval quality depends heavily on chunking strategy and will need iteration.
• MVP handles a single document at a time.

---

Deployment note

The proposal specified hosting on Hugging Face Spaces. Hugging Face has since moved free Gradio Space hosting behind a PRO subscription — only Static Spaces remain free, and organisation-owned Gradio Spaces require a Team plan. This repository therefore serves as the version-controlled source of record, and the application runs locally, which the assignment brief requires in any case. See the design document for the full rationale.