VLM Binary Classification Gap Analysis

VLM Binary Classification Gap Analysis

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/​FN failure cases to a JSONL file with a summary report. Run it with a TAO Data Services spec file; the data-services entrypoint requires -e <spec>.

Purpose

After running a VLM on a binary yes/​no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

Usage

Generate a vlm_bcq_spec.yaml with the bundled helper:

python3 skills/​data/​tao-analyze-gaps-vlm-bcq/​scripts/​prepare_vlm_bcq_spec.py \
  --predictions-json /​path/​to/​results.json \
  --videos-dir /​path/​to/​videos/​root \
  --results-dir /​path/​to/​output/​gaps \
  --output-spec /​path/​to/​output/​gaps/​vlm_bcq_spec.yaml

Omit --videos-dir when prediction video_id values are already absolute. The generated spec has this shape:

predictions_json: /​path/​to/​results.json
videos_dir: ""
results_dir: /​path/​to/​output/​gaps

Set videos_dir when video_id values in the predictions are relative paths:

predictions_json: /​path/​to/​results.json
videos_dir: /​path/​to/​videos/​root
results_dir: /​path/​to/​output/​gaps

Invoke the vlm_bcq action inside the TAO Toolkit data services container with -e <spec>:

gap_analysis vlm_bcq -e /​path/​to/​vlm_bcq_spec.yaml

Request exactly one GPU from the selected platform (compute_shape.gpus: 1, compute_shape.nodes: 1). VLM BCQ gap analysis does not perform GPU compute, but the Data Services image always calls nvidia-smi and fails when no GPU is visible. One is a GPU count, not a device ID; the platform selects the device.

After the run, surface the FP/​FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.

Inputs

• config spec: YAML file passed with -e. Template: assets/​default_vlm_bcq.yaml.
• predictions_json: Path to predictions JSON file. Must be a JSON array where each item has video_id, response, and gt fields. response and gt are parsed with word-boundary matching — 'yes' or 'no' anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
• videos_dir (optional): Base directory for resolving relative video_id paths. If omitted, video_id values are used as absolute paths.
• results_dir: Output directory for gap-analysis artifacts.

Predictions JSON format:

[
  {
    "video_id": "/​path/​to/​video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

Outputs

• kpi_gaps.jsonl: One JSON object per line for each FP/​FN case. Fields: video_id (absolute path), error_type (FP or FN), question, ground_truth, response.
• kpi_gaps_report.txt: Human-readable table with total FP/​FN counts.

If no gaps are found, no files are written and a message is logged.

Spec Fields

Keep the spec file and every path it references under the bind-mounted workspace so they resolve inside the container. Pass -e <spec> even if you also add Hydra overrides; current TAO Data Services entrypoints hard-require an experiment spec file before processing overrides.

Error Patterns