Skip to content
rag-eval logo

On-disk RAG evaluation (corpus/ + train.json)

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

nvidia/skills2.3Kinstalls3.5Kstars

SKILL.md

Full skill instructions

On-disk RAG evaluation (corpus/ + train.json)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/​eval/​evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/​stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/​rag-perf, docs/​performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/​alerting; or latency/​throughput benchmarking (use rag-perf).

Prerequisites

  • Repo cloned; run commands from repo root (imports and paths assume this).
  • Python 3.11+ and uv; eval deps: uv sync --project scripts/​eval.
  • Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
  • NVIDIA_API_KEY for RAGAS (see credential hygiene); optional RAG_EVAL_JUDGE_MODEL.
  • Dataset roots passed to --dataset-paths each contain corpus/ and train.json.

Instructions

  1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in references/​dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/​JSONL/​etc. using the patterns there.
  2. Run eval — uv run --project scripts/​eval python scripts/​eval/​evaluate_rag.py with --dataset-paths, --host, and --port. See references/​benchmark-execution.md for command examples, outputs, and errors. Use references/​evaluate-rag-cli.md for flag-level detail.
  3. Tune quality — Adjust --top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in references/​benchmark-execution.md when comparing retrieval/​generation configs for RAGAS scores.
  4. Analyze results — Use references/​result-analysis.md for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics.
  5. Triage errors — Use the error signal table and the Troubleshooting section below.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: references/​benchmark-execution.md#credential-hygiene-nvidia_api_key.

Minimal eval (key already in environment):

uv sync --project scripts/​eval
uv run --project scripts/​eval python scripts/​eval/​evaluate_rag.py \
  --dataset-paths /​path/​to/​my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/​my_dataset/​rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): references/​benchmark-execution.md.

Limitations

  • Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
  • Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
  • Scores depend on retrieval quality, judge model availability, and NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).
  • Large procedural detail lives under references/ to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Error / signalLikely causeWhat to do
Immediate exit mentioning NVIDIA_API_KEYMissing or invalid keySet key via secure channel; see credential hygiene in references/​benchmark-execution.md.
train.json must be a JSON arrayWrong JSON shapeTop-level array of objects; validate per references/​dataset-and-conversion.md.
Fewer rows in evaluation_data.json than train.jsonPer-query failuresCheck stderr: network or stream JSON errors; see error table in benchmark-execution.
Empty generated_contexts everywhereRetrieval gapVerify collection, ingestion, top_k / vdb_top_k, and ingestor_server_url without /​v1 suffix.
Ingestor 404 on uploadBad ingestor base URLPass http://host:port only — code appends /​v1/.

Full signal table: references/​benchmark-execution.md#common-error-cases-and-signals.

Gotchas

  • Run from repo root: paths and imports in scripts/​eval/​evaluate_rag.py assume this; a wrong directory silently breaks imports.
  • --ingestor_server_url: pass http://host:port without /​v1—the code appends /​v1/ automatically. Including /​v1 causes 404s on ingestor calls.
  • Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APP_VECTORSTORE_URL, embedding model).
  • --model / --llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
  • Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
  • Empty context metrics: if all generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

PieceLocation
Driverscripts/​eval/​evaluate_rag.py (CORPUS_DIRECTORY = corpus, EVAL_DATA = train.json)
Human README (always in-repo)scripts/​eval/​README.md
Full CLI (flags, defaults)scripts/​eval/​evaluate_rag.py --help; references/​evaluate-rag-cli.md
Dataset / conversionreferences/​dataset-and-conversion.md
Runs, outputs, errorsreferences/​benchmark-execution.md
Result analysis scriptsreferences/​result-analysis.md
Latency / throughputrag-perf skill, docs/​performance-benchmarking.md

Agent playbook

  1. Run eval — uv sync --project scripts/​eval then uv run --project scripts/​eval python scripts/​eval/​evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.
  2. Quality tuning — See references/​benchmark-execution.md: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.
  3. Data conversion — Follow references/​dataset-and-conversion.md.
  4. Analyze results — references/​result-analysis.md; quick scan: python3 -m json.tool results/<dataset>/​rag_<dataset>_evaluation_summary.json.
  5. Error triage — references/​benchmark-execution.md#common-error-cases-and-signals.

More skills from nvidia

cudaq-guide logo
nvidia/skills

cudaq-guide

CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.

3.5K 2.5K
View
data-designer logo
nvidia/skills

data-designer

Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline.

3.5K 2.4K
View
deepstream-dev logo
nvidia/skills

deepstream-dev

NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.

3.5K 2.4K
View
nemo-retriever logo
nvidia/skills

nemo-retriever

Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (`.jpg` `.png` `.tiff`), Office (`.docx` `.pptx`), text (`.html` `.txt`), audio (`.mp3` `.wav` `.m4a`), or video (`.mp4` `.mov`). Prefer this over native Read / Grep for multi-file or non-PDF corpora. Not for: editing files, web browsing, single-file plain-text lookups, fine-tuning.

3.5K 2.3K
View

Coordinate the end-to-end CAD/source-asset to SimReady workflow. Use for broad requests such as CAD to SimReady, source asset to simulation-ready USD, or prop packaging that require conversion, material/physics assignment, SimReady conformance, validation, and optional package creation; deploy or verify Content Agents services first when property assignment is enabled; route single-stage work through nested references.

3.5K 2.3K
View

Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.

3.5K 2.3K
View

Top-level workflow skill for USD performance diagnosis and optimization. Handles slow loading, high memory, low FPS, and broad scene-optimization requests; delegates auth/runtime setup to Phase 0 owners.

3.5K 2.3K
View

Vehicle routing (VRP, TSP, PDP) with cuOpt — Python API only. Use when the user is building or solving routing in Python.

3.5K 2.2K
View

Use only to generate or update a governance skill card for a specified existing agent skill directory. Do not use for explaining, listing, comparing, or discussing skill capabilities.

3.5K 2.2K
View

Use as the top-level router for Omniverse Realtime Viewer USD app requests and focused viewer reference documents.

3.5K 2.2K
View
rag-perf logo
nvidia/skills

rag-perf

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

3.5K 2.1K
View

cuOpt REST server — start server, endpoints, Python/curl client examples. Use when the user is deploying or calling the REST API.

3.5K 2.1K
View

Popular AI tools

Kaiber logo
Video

Kaiber

Generate, edit, and beat-sync AI video with leading models in one workspace.

Paid
View
Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View

Transform Your Design with AI Designer by ImgCreator.ai

Freemium
View
Akool AI logo
Content & writing

Akool AI

Revolutionizing Video Production with AI-Powered Creativity

Paid
View

Extend an image past the frame and let AI fill the new aspect ratio.

Freemium
View
StarByFace logo
Security

StarByFace

Discover your celebrity doppelgänger with StarByFace!

Free
View
C

ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.

Freemium
View
Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View