video-uq-failure-mode-benchmarks

Video-UQ Failure-Mode Benchmarks

This directory contains controlled MCQ benchmarks for diagnosing four VideoLLM failure modes: language-prior reliance, frame-sampling failure, evidence-extraction failure, and temporal insensitivity. The canonical release contains MVP, KFS-Bench, and the intervention-focused TempCompass counterfactual set. TVBench and the older mixed 300-item temporal subset are not part of the current dataset.

Failure labels are based on greedy answers only. The MCQ prompt shows the available options and asks for the option letter. Any stochastic generations retained from earlier baseline runs are intended for later uncertainty analysis and do not affect failure labels.

Canonical Composition

The KFS media folder also contains 121 Video-MME subtitle files. These are auxiliary metadata, not additional videos.

Layout

failure_mode_benchmarks/
  README.md
  language_prior/
    mvp.py
    mvp_balanced_300/
  sampling_sensitivity/
    create_kfs_subset.py
    get_kfs_subset_videolist.py
    extract_kfs_selected_videos.py
    kfs_balanced_300/
  temporal_ordering/
    create_tempcompass_counterfactuals.py
    download_tempcompass_counterfactuals.py
    tempcompass_official/
    tempcompass_counterfactuals_309/

Common Evaluation Process

1. Run each model with the same MCQ prompt and greedy decoding.
2. Resolve the prediction to an option index or canonical option text. Invalid-format responses count as incorrect.
3. Compare answers across controlled videos, visual conditions, frame samplers, or temporal counterfactual members.
4. Assign a failure label only when all interventions required by that label exist.
5. Retain the question, options, reference, generated answer, correctness, sampled frame indices, model, and intervention metadata in every result table.

The five current models are Qwen2.5-VL-7B, LLaVA-NeXT-Video-7B, InternVL3-8B, Qwen3-VL-32B, and InternVL3-38B.

Greedy MCQ Baselines

These figures were recomputed from the canonical video-conditioned Uniform-32 CSVs. The temporal column uses only the new 309-member TempCompass counterfactual set.

These are member-level accuracies, not failure-label rates. Sampling and evidence labels require oracle comparisons, while language-prior labels require no-video answers.

Language Prior: MVP Balanced 300

Path: language_prior/mvp_balanced_300/

Each MVP pair contains two minimally different videos with the same question and options but different correct answers. The benchmark compares three conditions for each pair and model:

Video A MCQ answer
Video B MCQ answer
No-video MCQ answer

Balance

Expected Labels

For MCQ labeling, answers are compared by the selected canonical option text rather than by letter alone because option order can differ between rows.

Files

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/language_prior
python mvp.py --total-pairs 300 --seed 42 --output-dir mvp_balanced_300

cd /home/debarpanb/videouq/failure_mode_benchmarks
python download_selected_videos_from_hf.py --suite mvp

Frame Sampling and Evidence Extraction: KFS Balanced 300

Path: sampling_sensitivity/kfs_balanced_300/

KFS provides required evidence scenes and temporal segments. Every practical sampler uses the same model, question, MCQ options, preprocessing, and frame budget.

Balance

There is at most one selected question per underlying video. The annotation table has 951 required evidence segments.

Sampling Process

BSR=1 means every required scene receives its required frame allocation. BSR<1 means at least one required scene is missing or underrepresented.

Expected Labels

Oracle-64 is a secondary budget diagnostic. It does not replace Oracle-32 when distinguishing same-budget sampling from evidence-extraction failure.

Files

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/sampling_sensitivity
python create_kfs_subset.py --total 300 --seed 42 --output-dir kfs_balanced_300
python get_kfs_subset_videolist.py
python extract_kfs_selected_videos.py

Temporal Grounding: TempCompass Counterfactuals 309

Path: temporal_ordering/tempcompass_counterfactuals_309/

This set contains complete controlled groups from the official TempCompass processing metadata. Every member in a group uses the same MCQ question, while the video and correct answer change according to the temporal intervention.

Counterfactual Composition

The 309 members resolve to 301 unique videos. Four reverse-video bases participate in both a direction and an attribute-change question group.

Counterfactual Construction

• Direction and attribute groups pair an official original video with its _reverse version.
• Relative-speed groups use the official _concat_0, _concat_1, and _concat_2 videos and metadata to identify same-speed, slower, and faster conditions.
• Event-order groups use the official concatenations for simultaneous events, event A before B, and event B before A.
• The builder selects one question stem shared by every member and verifies that the authoritative answers distinguish the required conditions.
• Exactly 307 members use an official shared question. Two direction members use one metadata-derived canonical stem because the official wording changes from appear to disappear after reversal.

Expected Labels

MCQ equivalence compares selected canonical option text, not raw letters. A changed letter alone is not evidence of temporal sensitivity when option order changes.

Files

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/temporal_ordering
python create_tempcompass_counterfactuals.py

source /home/debarpanb/miniconda3/etc/profile.d/conda.sh
conda activate videouq
python download_tempcompass_counterfactuals.py

To regenerate only resolved annotations and manifests from existing videos:

python download_tempcompass_counterfactuals.py --skip-download

Common Fields

• question: natural-language question stem.
• question_with_options: question plus MCQ options.
• candidates or candidates_json: ordered answer options.
• answer or answer_text: authoritative correct option text.
• answer_index: zero-based correct option index.
• answer_letter: corresponding option letter.
• video_path and related video identifiers: required media reference.
• counterfactual_group_id, counterfactual_intervention, and counterfactual_condition: temporal grouping metadata.
• required_segments and required_scene_ids: KFS evidence annotations.

Prefer JSONL for evaluation code and CSV for manual inspection. Summary JSON files provide the quickest integrity check for the intended composition.

TempCompass is distributed for academic research under CC BY-NC 4.0. Preserve the original attribution and license conditions when redistributing this derivative set.