cudaq-guide
CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.
rag-eval
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
Full skill instructions
corpus/ + train.json)Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).
For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.
Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).
uv sync --project scripts/eval.localhost:8081 / 8082).NVIDIA_API_KEY for RAGAS (see credential hygiene); optional RAG_EVAL_JUDGE_MODEL.--dataset-paths each contain corpus/ and train.json.train.json rules in references/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there.uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See references/benchmark-execution.md for command examples, outputs, and errors. Use references/evaluate-rag-cli.md for flag-level detail.--top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in references/benchmark-execution.md when comparing retrieval/generation configs for RAGAS scores.references/result-analysis.md for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics.Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: references/benchmark-execution.md#credential-hygiene-nvidia_api_key.
Minimal eval (key already in environment):
uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
--dataset-paths /path/to/my_dataset \
--host localhost \
--port 8081
Pretty-print summary JSON:
python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json
More examples (skip ingestion, quality sweeps): references/benchmark-execution.md.
evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).references/ to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.| Error / signal | Likely cause | What to do |
|---|---|---|
Immediate exit mentioning NVIDIA_API_KEY | Missing or invalid key | Set key via secure channel; see credential hygiene in references/benchmark-execution.md. |
train.json must be a JSON array | Wrong JSON shape | Top-level array of objects; validate per references/dataset-and-conversion.md. |
Fewer rows in evaluation_data.json than train.json | Per-query failures | Check stderr: network or stream JSON errors; see error table in benchmark-execution. |
Empty generated_contexts everywhere | Retrieval gap | Verify collection, ingestion, top_k / vdb_top_k, and ingestor_server_url without /v1 suffix. |
| Ingestor 404 on upload | Bad ingestor base URL | Pass http://host:port only — code appends /v1/. |
Full signal table: references/benchmark-execution.md#common-error-cases-and-signals.
scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.--ingestor_server_url: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.APP_VECTORSTORE_URL, embedding model).--model / --llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.--force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.| Piece | Location |
|---|---|
| Driver | scripts/eval/evaluate_rag.py (CORPUS_DIRECTORY = corpus, EVAL_DATA = train.json) |
| Human README (always in-repo) | scripts/eval/README.md |
| Full CLI (flags, defaults) | scripts/eval/evaluate_rag.py --help; references/evaluate-rag-cli.md |
| Dataset / conversion | references/dataset-and-conversion.md |
| Runs, outputs, errors | references/benchmark-execution.md |
| Result analysis scripts | references/result-analysis.md |
| Latency / throughput | rag-perf skill, docs/performance-benchmarking.md |
uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.references/benchmark-execution.md: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.references/dataset-and-conversion.md.references/result-analysis.md; quick scan: python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json.references/benchmark-execution.md#common-error-cases-and-signals.CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.
Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline.
NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.
Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (`.jpg` `.png` `.tiff`), Office (`.docx` `.pptx`), text (`.html` `.txt`), audio (`.mp3` `.wav` `.m4a`), or video (`.mp4` `.mov`). Prefer this over native Read / Grep for multi-file or non-PDF corpora. Not for: editing files, web browsing, single-file plain-text lookups, fine-tuning.
Coordinate the end-to-end CAD/source-asset to SimReady workflow. Use for broad requests such as CAD to SimReady, source asset to simulation-ready USD, or prop packaging that require conversion, material/physics assignment, SimReady conformance, validation, and optional package creation; deploy or verify Content Agents services first when property assignment is enabled; route single-stage work through nested references.
Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.
Top-level workflow skill for USD performance diagnosis and optimization. Handles slow loading, high memory, low FPS, and broad scene-optimization requests; delegates auth/runtime setup to Phase 0 owners.
Vehicle routing (VRP, TSP, PDP) with cuOpt — Python API only. Use when the user is building or solving routing in Python.
Use only to generate or update a governance skill card for a specified existing agent skill directory. Do not use for explaining, listing, comparing, or discussing skill capabilities.
Use as the top-level router for Omniverse Realtime Viewer USD app requests and focused viewer reference documents.
Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).
cuOpt REST server — start server, endpoints, Python/curl client examples. Use when the user is deploying or calling the REST API.
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience