Skip to content
rag-perf logo

RAG-Perf — config-driven perf benchmark CLI

rag-perf

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

nvidia/skills2.1Kinstalls3.5Kstars

SKILL.md

Full skill instructions

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

  • Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
  • Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
  • Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
  • Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

  • Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
  • Python 3.11+ and uv on PATH.
  • Install rag-perf into its own uv-managed venv: uv sync --project scripts/​rag-perf.
  • For unit tests: install dev extras as well — uv sync --project scripts/​rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
  • A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./​scripts/​rag-perf registers it via the aiperf.plugins entry point.
  • For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/​v1/​chat/​completions).
  • rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

  1. Pick a preset. The three under scripts/​rag-perf/​configs/ are:

    • quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
    • single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
    • sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).
  2. Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /​v1/​collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.

  3. Run. From repo root:

    uv run --project scripts/​rag-perf rag-perf -c scripts/​rag-perf/​configs/​single_run.yaml
    

    Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.

  4. Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See references/​output-and-analysis.md.

  5. Inspect artifacts. Layout depends on run shape — flat for single-point + iterations=1, nested under iter_<i>/<point>/​... otherwise. See references/​output-and-analysis.md for the full directory tree, file purposes, and how to parse results.json / results.csv / report.md.

  6. Summarise for the user. When reporting back, follow the playbook in references/​output-and-analysis.md#summarising-results-to-the-user: pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, always flag zero citations / non-zero error rate / suspect llm_ttft_ms / small-sample p99, and propose a concrete next-experiment YAML.

  7. Tune. Schema is fully documented in docs/​performance-benchmarking.md and the deeper-dive references below. Common knobs: turn aiperf.enabled: false for profile-only mode, increase load.iterations for variance estimation, set load.sleep_between_points_s: 60 for overnight Cartesian sweeps.

Examples

Profile-only (quickest signal on retrieval / reranker tuning):

uv run --project scripts/​rag-perf rag-perf -c scripts/​rag-perf/​configs/​quick_profile.yaml

Output: rag-perf-results/​quick_profile/​run_<ts>/​{profile_report.md, profile_results.json, profiling/​}. The aiperf_rag_on/ directory is omitted. Filenames are profile_* because aiperf.enabled: false.

Single benchmark point with full report:

uv run --project scripts/​rag-perf rag-perf -c scripts/​rag-perf/​configs/​single_run.yaml

Output: flat run_<ts>/​{report.md, results.json, results.csv, profiling/, aiperf_rag_on/​}.

Concurrency sweep:

uv run --project scripts/​rag-perf rag-perf -c scripts/​rag-perf/​configs/​sweep.yaml

Output: nested run_<ts>/​iter_1/<CR:_VDB-K:_RERANKER-K:_…>/​{profiling,aiperf_rag_on}/ per point, plus aggregate report.md / results.json / results.csv at the run root.

Run unit tests:

uv sync --project scripts/​rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/​rag-perf python -m pytest tests/​unit/​test_rag_perf/

Limitations

  • The CLI is config-only: author or copy YAML to vary a parameter.
  • load.concurrency / rag.vdb_top_k / rag.reranker_top_k accept int | list[int]; the validator requires unique list values because each value names a unique point dir.
  • input.file and input.synthetic follow an XOR rule — both set fails validation. When neither is set, synthetic auto-fills with defaults so a bare config still validates.
  • File-based input format is inferred from extension only (.jsonl or .csv); other extensions are rejected.
  • Synthetic generation streams each query to disk as it completes (failure-resilient) but fails fast on the first LLM error — partial JSONL is preserved. Re-run after fixing the endpoint.
  • Reasoning models (Nemotron Omni, Qwen-Reasoning) require synthetic.disable_thinking: true (the default). Without it the model exhausts the token budget on chain-of-thought and content returns empty — the generator now raises with a clear message instead of substituting reasoning_content for the answer.
  • aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing AiperfRunner._base_aiperf_cmd in scripts/​rag-perf/​rag_perf/​runner.py.
  • Procedural detail lives under references/ to keep this file concise.

Troubleshooting

Error / signalLikely causeWhat to do
Configuration errors in <yaml>: • input — ... XOR ruleBoth input.file and input.synthetic setPick one. The XOR validator runs at YAML load time.
input.file must end in .jsonl or .csvExtension other than .jsonl / .csvRename or convert.
load.concurrency has duplicate valuese.g. [2, 2, 4]Each concurrency maps to a unique point dir; dedupe.
warmup_requests must be >= 1YAML had warmup_requests: 0aiperf rejects warmup=0; minimum is 1.
LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).Reasoning model used CoT and ran out of tokensSet synthetic.disable_thinking: true (the default) or raise min_query_tokens.
✗ All N profiling requests failed across M point(s). + exit 1Bad URL, server down, wrong collectionVerify target.url, rag.collection_names (the <collection_name> placeholder will hit this).
Per-iteration ⚠ N profiling requests failed warning, run continuesSome requests timed out / errored mid-runCheck rag-server logs, raise target.timeout_s, drop concurrency.
RuntimeError: Random synthetic query generation failed at query N: ...LLM endpoint rejected a request mid-generationPartial JSONL is at synthetic.jsonl_output_path; fix endpoint and re-run with reduced num_queries, or point input.file at the partial file.
Citation count (mean): 0 and Citation relevance score: N/​A for a non-empty deploymentCollection mismatch between rag.collection_names and what's actually ingestedRun curl -s http://<ingestor>:8082/​v1/​collections to list real collections.
Tests error with ModuleNotFoundError: No module named 'pytest_asyncio'Dev extras missinguv sync --project scripts/​rag-perf --extra dev.
CI: ModuleNotFoundError: No module named 'ruamel' from tests/​unit/​test_rag_perf/rag-perf package missing from CI venvAdd uv pip install -e ./​scripts/​rag-perf after the top-level install in the unit-tests job.

Gotchas

  • Run from repo root. Preset configs reference scripts/​rag-perf/​examples/​queries.jsonl and scripts/​rag-perf/​prompts/​default_prompts.yaml with repo-root-relative paths. Running from inside scripts/​rag-perf/ will fail those file lookups.
  • CLI is config-only. Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.
  • Always edit rag.collection_names before the first run. The presets ship with ["<collection_name>"] as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as Citation count (mean): 0 everywhere.
  • load.concurrency_list, rag.vdb_top_k_list, rag.reranker_top_k_list are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.
  • aiperf.enabled: false changes filenames. The top-level outputs become profile_report.md / profile_results.json / profile_results.csv. The aggregate sweep table also suppresses load-test rows and the "Optimal throughput" footer.
  • Resolved-config dump is verbose (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.
  • The aiperf shell command is logged before each subprocess. Look for \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ... in stdout — copy-paste runnable for reproducing a single point outside rag-perf.
  • --endpoint-type nvidia_rag comes from the bundled plugin at scripts/​rag-perf/​rag_perf/​plugin/​nvidia_rag.py. It teaches aiperf about the RAG /​v1/​generate request shape and parses citations + per-stage metrics out of the SSE stream. If aiperf can't resolve nvidia_rag, rag-perf needs editable installation in the venv — re-run uv sync --project scripts/​rag-perf (or uv pip install -e ./​scripts/​rag-perf).
  • Sweep-mode point-name collision. When two points differ only in concurrency (e.g. [1, 4] × single vdb_top_k), the dir name encodes everything: CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:.... Cluster / GPU / experiment_name (output.cluster, output.gpu, output.experiment_name) are appended too — useful for diff-friendly artifact paths across machines.
  • load.iterations > 1 repeats the entire grid. Each repetition writes to its own iter_<i>/. Aggregate CSV row count = n_points × iterations.

Source of truth

Agent playbook

  1. Sync deps: uv sync --project scripts/​rag-perf (one-time per checkout).
  2. Pick & customise a preset: copy scripts/​rag-perf/​configs/<preset>.yaml if you want a variant; always set rag.collection_names to a real collection.
  3. Run: uv run --project scripts/​rag-perf rag-perf -c <config> from repo root.
  4. Read the per-point + aggregate tables on stdout. Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.
  5. Parse artifacts under output.dir/​run_<ts>/ — see references/​output-and-analysis.md. For multi-point runs, results.csv has one row per (point × iteration).
  6. Summarise for the user using the playbook in references/​output-and-analysis.md#summarising-results-to-the-user — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations / non-zero errors / suspect llm_ttft_ms / low sample size, and a concrete next-experiment YAML.
  7. Tune retrieval / reranker: flip to quick_profile.yaml or aiperf.enabled: false for fast iteration, then return to single_run.yaml / sweep.yaml when characterising under load.
  8. Triage failures: see Troubleshooting above and references/​output-and-analysis.md for empty-citation / bottleneck=N/​A patterns.

More skills from nvidia

cudaq-guide logo
nvidia/skills

cudaq-guide

CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.

3.5K 2.5K
View
data-designer logo
nvidia/skills

data-designer

Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline.

3.5K 2.4K
View
deepstream-dev logo
nvidia/skills

deepstream-dev

NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.

3.5K 2.4K
View
rag-eval logo
nvidia/skills

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

3.5K 2.3K
View
nemo-retriever logo
nvidia/skills

nemo-retriever

Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (`.jpg` `.png` `.tiff`), Office (`.docx` `.pptx`), text (`.html` `.txt`), audio (`.mp3` `.wav` `.m4a`), or video (`.mp4` `.mov`). Prefer this over native Read / Grep for multi-file or non-PDF corpora. Not for: editing files, web browsing, single-file plain-text lookups, fine-tuning.

3.5K 2.3K
View

Coordinate the end-to-end CAD/source-asset to SimReady workflow. Use for broad requests such as CAD to SimReady, source asset to simulation-ready USD, or prop packaging that require conversion, material/physics assignment, SimReady conformance, validation, and optional package creation; deploy or verify Content Agents services first when property assignment is enabled; route single-stage work through nested references.

3.5K 2.3K
View

Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.

3.5K 2.3K
View

Top-level workflow skill for USD performance diagnosis and optimization. Handles slow loading, high memory, low FPS, and broad scene-optimization requests; delegates auth/runtime setup to Phase 0 owners.

3.5K 2.3K
View

Vehicle routing (VRP, TSP, PDP) with cuOpt — Python API only. Use when the user is building or solving routing in Python.

3.5K 2.2K
View

Use only to generate or update a governance skill card for a specified existing agent skill directory. Do not use for explaining, listing, comparing, or discussing skill capabilities.

3.5K 2.2K
View

Use as the top-level router for Omniverse Realtime Viewer USD app requests and focused viewer reference documents.

3.5K 2.2K
View

cuOpt REST server — start server, endpoints, Python/curl client examples. Use when the user is deploying or calling the REST API.

3.5K 2.1K
View

Popular AI tools

Kaiber logo
Video

Kaiber

Generate, edit, and beat-sync AI video with leading models in one workspace.

Paid
View
Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View

Transform Your Design with AI Designer by ImgCreator.ai

Freemium
View
Akool AI logo
Content & writing

Akool AI

Revolutionizing Video Production with AI-Powered Creativity

Paid
View

Extend an image past the frame and let AI fill the new aspect ratio.

Freemium
View
StarByFace logo
Security

StarByFace

Discover your celebrity doppelgänger with StarByFace!

Free
View
C

ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.

Freemium
View
Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View