StressRAG-Artifacts

StressRAG ISSTA 2026 Experiments

For the paper "StressRAG: Finding Brittle Queries in RAG through Evaluator-Aligned and Budget-Aware Test Selection", This reposotey runs evaluation suites for a retrieval-augmented generation (RAG) system and compares selection strategies (StressRAG, ARES, RAGAS, Random) on two datasets (TriviaQA and LegalBench). It builds a FAISS vector index, retrieves documents, generates answers with a local Ollama model, and logs retrieval + generation metrics per query and per suite.

What's in here

• main.py: experiment runner (selects suites, runs RAG, logs metrics).
• baselines.py: ARES and RAGAS selection baselines.
• evaluators.py: retrieval and generation metrics.
• utils.py: dataset loading + helper utilities.
• data/: datasets and corpora.

Requirements

• Python 3.10+ recommended.
• Local Ollama server running (for generation and the weak agent model).
• OpenAI API key (for the strong agent model).

Install dependencies:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Optional (improves text normalization quality in evaluators):

python -m spacy download en_core_web_sm

Data layout

The loader expects the following files:

data/
  LegalBench/
    legal_data.json
    legal_data_corpus.json
  TriviaQA/
    trivia_data.json
    trivia_data_corpus.json

Configuration (edit main.py)

Key knobs at the top of main.py:

• DATASET_NAME: "legalbench" or "triviaqa"
• GEN_MODEL: Ollama model used for answer generation (default phi3:mini)
• STRONG_AGENT_MODEL: OpenAI model for strong agent (default gpt-5-nano)
• EMBEDDING_MODEL_ID: sentence-transformers embedding model
• COMPARISON_BASELINES: which strategies to run

Running the experiment

1. Start Ollama and ensure the models are pulled:

ollama serve
ollama pull phi3:mini

1. Set your OpenAI key (needed for STRONG_AGENT_MODEL):

setx OPENAI_API_KEY "your_key_here"

1. Run:

python main.py

Outputs

The run creates a timestamped results folder:

issta_results_2026_<dataset>/
  issta_suite_metrics_<timestamp>.csv
  issta_query_details_<timestamp>.csv
  experiment_metadata_<timestamp>.json
  suite_logs_<seed>_<strategy>_<timestamp>.txt

It also creates or reuses a FAISS index under (if does not exist):

vector_store_mxbai_<dataset>/

Notes

• If issta_retrieval_cache_<dataset>.json exists in the repo root, it will be used to speed up retrieval scoring. Otherwise, the run will proceed without it (slower).
• If you don't want to use OpenAI, remove StressRAG from COMPARISON_BASELINES or switch to StressRAG-NO-AGENT (also called StressRAG-Lite).