Skip to content
deepstream-import-vision-model logo

DeepStream Import Vision Model

deepstream-import-vision-model

Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.

nvidia/skills2.3Kinstalls3.5Kstars

SKILL.md

Full skill instructions

DeepStream Import Vision Model

When this skill is active, read the relevant reference document before starting each phase. Do not rely on memory — reference documents contain exact script paths, bash variable conventions, log filename contracts, and critical parsing rules.

Current scope: Object detection models only. Fail fast on classification, segmentation, or other architectures detected in config.json.

Model choice — always offer two options

Before preflight, browsing, downloads, or file creation, present exactly these two choices. Do not start with only an open-ended model-source prompt. If the user's request already clearly selects a model, confirm the matching choice instead of asking redundantly.

1. Default model (recommended)

Use the validated Hugging Face RT-DETR model:

model_id: PekingU/​rtdetr_r50vd
source: huggingface
task: object-detection
precision_preference: fp16

2. Custom object-detection model

Ask for one supported source:

  • Hugging Face model ID (organization/​model) or full model URL.
  • NVIDIA NGC catalog model URL including its version.

Explain that the skill currently rejects classification, segmentation, and other non-detection architectures after inspecting config.json. Do not invent or silently substitute a model when the custom source is missing or unsupported.

For a dry run, present the same two choices and simulate discovery, build, benchmark, and report stages without browsing, downloading, launching Docker, writing files, or starting processes.

Pipeline Overview

StepPhaseReferenceWhat it does
1–3Model Acquirereferences/​model-acquire.mdBrowse HF/​NGC, detect format, download ONNX or export SafeTensors
4–5Engine Buildreferences/​engine-build.mdBuild dynamic TRT engine, run trtexec BS=1 and BS=MAX_BS
6–7DS Pipelinereferences/​pipeline-run.mdCustom bbox parser, nvinfer config, single-stream + multi-stream benchmarks
8Reportreferences/​report-generation.md5 charts, HTML, PDF benchmark report

Run the full pipeline autonomously without pausing for confirmation at each step.

Runs entirely through Docker (no host packages)

Every step runs INSIDE the DeepStream container. The host needs only Docker + the NVIDIA driver — no host python/​venv/​torch/​trtexec/​make/​wkhtmltopdf. This works identically on Linux and Windows (Docker Desktop + WSL2 backend, required for --gpus). The per-shell bind-mount token is the only OS difference — -v "$PWD":/​work (bash), -v "${PWD}:/​work" (PowerShell), -v "%cd%:/​work" (cmd); full guide in references/​windows.md. All venv/​ONNX/ engine/​parser/​config/​report artifacts live under the mounted working root and persist between the ephemeral --rm containers.

Pre-flight — bootstrap + verify (through the container)

1. One-time bootstrap — builds build/​.venv_optimum (torch/​onnx/​onnxruntime/​report deps; the venv name is historical, optimum is no longer used) + installs wkhtmltopdf, all in-container. From the working root:

docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/​work -w /​work \
  --entrypoint bash nvcr.io/​nvidia/​deepstream:9.1-triton-multiarch \
  .claude/​skills/​deepstream-import-vision-model/​setup.sh

2. Preflight — GPU + venv + trtexec, run THROUGH the container (container-mode auto-detects):

docker run --rm --gpus all -v "$PWD":/​work -w /​work \
  --entrypoint bash nvcr.io/​nvidia/​deepstream:9.1-triton-multiarch \
  .claude/​skills/​deepstream-import-vision-model/​scripts/​preflight.sh   # proceed only on PASS

Every subsequent phase runs the same way — issue the model's commands via docker run … --entrypoint bash … -lc '<commands>' (or the .claude/​skills/​deepstream-import-vision-model/​scripts/​dsrun.sh wrapper: bash .claude/​skills/​deepstream-import-vision-model/​scripts/​dsrun.sh '<in-container command>'), using PY=build/​.venv_optimum/​bin/​python and trtexec at /​usr/​src/​tensorrt/​bin/​trtexec inside the container. deepstream-app, gst-launch-1.0, and /​opt/​nvidia/​deepstream/… sample paths all exist in the image. TensorRT build+runtime share one image, so there is no version skew (the concern the old "build on the host" rule tried to avoid — see references/​engine-build.md). sample_720p.mp4 ships in the image; set DS_VIDEO only to override.

Mandatory Output Structure

Create once MODEL_NAME is known (Step 1). Never dump files flat.

models/​{model_name}/
  model/           <- ONNX file(s)
  parser/          <- .cpp, Makefile, .so
  config/          <- nvinfer config, ds-app config, labels.txt
  scripts/         <- run helper scripts
  benchmarks/
    engines/       <- _dynamic_b{MAX_BS}.engine, timing.cache, build logs
    b1/            <- trtexec BS=1 log
    b{MAX_BS}/     <- trtexec BS=MAX_BS log
    ds/            <- DS benchmark logs
  reports/         <- benchmark_report.md, .html, .pdf, benchmark_data.json
    charts/        <- chart_*.png (5 charts)
  samples/         <- output .mp4 or .ogv (theoraenc fallback), test frames
    kitti_output/  <- KITTI detection .txt files
mkdir -p models/​$MODEL_NAME/​{model,parser,config,scripts,benchmarks/​engines,benchmarks/​ds,reports/​charts,samples/​kitti_output}

Critical Rules

  1. Engine naming — always {model}_dynamic_b{MAX_BS}.engine. Never bare model_dynamic.engine.
  2. batch_size == num_streams — in DS runs, batch-size and stream count are always equal.
  3. Log filenames are fixed — trtexec_b1.log, trtexec_b${MAX_BS}.log, ds_s${N}_run1.log, ds_s${N}_run2.log. No timestamps. Report generation reads exact paths.
  4. Parser zero-init — always NvDsInferObjectDetectionInfo obj = {};. Required for DS 9.1 OBB support; bare obj; leaves rotation_angle uninitialized, causing tilted bounding boxes.
  5. KITTI validation gate — do NOT proceed to Step 7 if KITTI frame count is zero or detection rate < 90%.
  6. Shared venv — build/​.venv_optimum reused across all models. Never create per-model venvs.
  7. trtexec --noDataTransfers — GPU-only compute matches DeepStream's GPU-to-GPU data flow.
  8. Report HTML+PDF — always use .claude/​skills/​deepstream-import-vision-model/​scripts/​report/​md-to-html-pdf.py. Never write a custom HTML generator or call wkhtmltopdf directly.
  9. Object detection only — reject non-detection architectures from config.json before building anything.
  10. Encoder fallback (MANDATORY) — x264enc and openh264enc are prohibited. On NVENC-unavailable systems, use theoraenc + oggmux (LGPL; ships in gst-plugins-base; output is .ogv). If theoraenc/oggmux are absent, skip video creation (DS_SINGLE_STREAM_MODE=skipped). Report which mode was used: nvv4l2h264enc / theoraenc-fallback / skipped.
  11. Video source (MANDATORY) — default is always sample_720p.mp4 (1280×720). Never autonomously substitute sample_1080p_h264.mp4 or any other file. Only use a different video when the user explicitly provides a path (via DS_VIDEO env var or script argument).

Examples

Default model, end to end. Bootstrap once, then run the full pipeline:

docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/​work -w /​work \
  --entrypoint bash nvcr.io/​nvidia/​deepstream:9.1-triton-multiarch \
  .claude/​skills/​deepstream-import-vision-model/​setup.sh
# then: "Use deepstream-import-vision-model to run PekingU/​rtdetr_r50vd"

SafeTensors model with no published ONNX. Step 2b exports it first; the wrapper reports which backend produced the graph and fails loudly if the batch dimension was baked in:

bash .claude/​skills/​deepstream-import-vision-model/​scripts/​model/​safetensors-to-onnx.sh \
  models/​$MODEL_NAME/​hf_model models/​$MODEL_NAME/​onnx_export/
#   [export] backend=dynamo
#   [export] dynamo produced a static batch dimension; trying the next backend
#   [export] backend=legacy-torchscript
#   [export] pixel_values shape=['batch', 3, 640, 640]

Pin a Hub revision for a reproducible build — any exporter flag passes straight through:

bash .claude/​skills/​deepstream-import-vision-model/​scripts/​model/​safetensors-to-onnx.sh \
  PekingU/​rtdetr_r50vd models/​rtdetr/​onnx_export --revision <commit-sha> --opset 18

Pipeline Timing

Wrap every step:

STEP_START=$(date +%s.%N)
# ... step commands ...
STEP_END=$(date +%s.%N)
STEP_DURATION=$(python3 -c "print(round($STEP_END - $STEP_START, 2))")   # bc is not in the container; python3 always is
echo "[Step N] completed in ${STEP_DURATION}s"

Track PIPELINE_START (before Step 1) and PIPELINE_END (after Step 8). Report all durations in the benchmark report.

Report Output (MANDATORY — all 3 formats)

  1. benchmark_report.md — markdown source (12 mandatory sections)
  2. benchmark_report.html — styled HTML (charts base64-inlined, no local file access)
  3. benchmark_report_{model_name}.pdf — via md-to-html-pdf.py; verify charts are embedded by counting data:image/​png occurrences in the HTML output: grep -o 'data:image/​png' benchmark_report.html | wc -l should equal 5

Run charts and report scripts with the shared venv active: source build/​.venv_optimum/​bin/​activate.

Reference Documents

IMPORTANT: Read the relevant reference before starting each phase. Do NOT generate code from memory.

DocumentUse When
references/​model-acquire.mdSteps 1–3: HF/​NGC URL parsing, format detection, ONNX download, SafeTensors export, label extraction
references/​engine-build.mdSteps 4–5: trtexec engine build, benchmarks, PEAK_GPU_STREAMS derivation, iterative scaling
references/​pipeline-run.mdSteps 6–7: custom bbox parser, nvinfer config, single-stream validation, KITTI dump, multi-stream benchmark
references/​report-generation.mdStep 8: benchmark_data.json, 5 charts, 12-section markdown report, HTML + PDF

Scripts

Installed into .claude/​skills/​deepstream-import-vision-model/​scripts/ by install.sh.

ScriptPhasePurpose
model/​hf-list-files.sh1–3List HuggingFace repo files
model/​hf-download-config.sh1–3Download config.json from HF
model/​ngc-list-files.sh1–3List NGC model files
model/​ngc-download.sh1–3Download NGC model archive
model/​safetensors-to-onnx.sh1–3Export SafeTensors → ONNX via torch.onnx.export (wrapper)
model/​safetensors_to_onnx.py1–3The exporter — dynamo backend, TorchScript fallback, verifies dynamic batch
model/​inspect-onnx.py1–5Inspect ONNX input/​output shapes
model/​make-static-batch-onnx.py4–5Bake batch dim into ONNX
model/​cleanup.shAnyRemove staging dirs, preserve shared venv
engine/​benchmark-trtexec.sh4–5Run trtexec with standard flags
deepstream/​ds-single-stream.sh6–7Single-stream visual validation (NVENC primary; theoraenc+oggmux fallback; skip if neither)
deepstream/​ds-sweep.sh6–72-phase batch size sweep
deepstream/​benchmark-ds.sh6–7Fixed-stream DS benchmark
deepstream/​ds-kitti-dump.sh6–7KITTI detection dump via deepstream-app
deepstream/​ds-perf-run.sh7Step 7c two-run benchmark — wraps deepstream-app with enable-perf-measurement=1, writes fixed-name log for the report parser
deepstream/​extract-frame.sh6–7Extract sample frames from output video (.mp4 NVENC path or .ogv theoraenc fallback)
report/​generate-benchmark-charts.py8Generate 5 benchmark PNG charts
report/​md-to-html-pdf.py8Markdown → styled HTML → PDF (canonical benchmark report path)
report/​md-to-pdf.shAnyMarkdown → PDF via pandoc/​pdflatex — for design docs and references only, NOT for benchmark reports (use md-to-html-pdf.py for those)
report/​report-style.css8CSS for HTML report
report/​render-mermaid-for-pdf.py8Mermaid diagram → PNG
report/​mermaid-puppeteer.json8Vetted Puppeteer config for Mermaid (sandboxed; non-root)
report/​mermaid-puppeteer-root.json8Vetted Puppeteer config for Mermaid (used when running as root)

Quick Error Reference

ErrorFix
Tilted/​diagonal bounding boxesParser struct not zero-initialized — use NvDsInferObjectDetectionInfo obj = {};
Zero KITTI filesgie-kitti-output-dir not read by nvinfer — use ds-kitti-dump.sh (wraps deepstream-app)
Engine rebuilds every DS runmodel-engine-file path wrong — check relative path from config/ dir
setDimensions negative dimsAdd infer-dims=3;H;W to nvinfer config for dynamic ONNX models
--memPoolSize workspace 0.03 MiBUse M suffix not MiB — e.g. --memPoolSize=workspace:32768M
ForeignNode build failure (DETR)Run onnxsim — see references/​engine-build.md. Not reproduced on TRT 10.16 with either export backend
ONNX has a static batch dimBoth export backends specialized it — see the gotchas in references/​model-acquire.md
Zero detectionsWrong net-scale-factor — check model family table in references/​pipeline-run.md
No module named 'pyservicemaker'Install into venv: pip install /​opt/​nvidia/​deepstream/​.../​pyservicemaker*.whl

More skills from nvidia

cudaq-guide logo
nvidia/skills

cudaq-guide

CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.

3.5K 2.5K
View
data-designer logo
nvidia/skills

data-designer

Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline.

3.5K 2.4K
View
deepstream-dev logo
nvidia/skills

deepstream-dev

NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.

3.5K 2.4K
View
rag-eval logo
nvidia/skills

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

3.5K 2.3K
View
nemo-retriever logo
nvidia/skills

nemo-retriever

Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (`.jpg` `.png` `.tiff`), Office (`.docx` `.pptx`), text (`.html` `.txt`), audio (`.mp3` `.wav` `.m4a`), or video (`.mp4` `.mov`). Prefer this over native Read / Grep for multi-file or non-PDF corpora. Not for: editing files, web browsing, single-file plain-text lookups, fine-tuning.

3.5K 2.3K
View

Coordinate the end-to-end CAD/source-asset to SimReady workflow. Use for broad requests such as CAD to SimReady, source asset to simulation-ready USD, or prop packaging that require conversion, material/physics assignment, SimReady conformance, validation, and optional package creation; deploy or verify Content Agents services first when property assignment is enabled; route single-stage work through nested references.

3.5K 2.3K
View

Top-level workflow skill for USD performance diagnosis and optimization. Handles slow loading, high memory, low FPS, and broad scene-optimization requests; delegates auth/runtime setup to Phase 0 owners.

3.5K 2.3K
View

Vehicle routing (VRP, TSP, PDP) with cuOpt — Python API only. Use when the user is building or solving routing in Python.

3.5K 2.2K
View

Use only to generate or update a governance skill card for a specified existing agent skill directory. Do not use for explaining, listing, comparing, or discussing skill capabilities.

3.5K 2.2K
View

Use as the top-level router for Omniverse Realtime Viewer USD app requests and focused viewer reference documents.

3.5K 2.2K
View
rag-perf logo
nvidia/skills

rag-perf

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

3.5K 2.1K
View

cuOpt REST server — start server, endpoints, Python/curl client examples. Use when the user is deploying or calling the REST API.

3.5K 2.1K
View

Popular AI tools

Kaiber logo
Video

Kaiber

Generate, edit, and beat-sync AI video with leading models in one workspace.

Paid
View
Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View

Transform Your Design with AI Designer by ImgCreator.ai

Freemium
View
Akool AI logo
Content & writing

Akool AI

Revolutionizing Video Production with AI-Powered Creativity

Paid
View

Extend an image past the frame and let AI fill the new aspect ratio.

Freemium
View
StarByFace logo
Security

StarByFace

Discover your celebrity doppelgänger with StarByFace!

Free
View
C

ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.

Freemium
View
Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View