ProactivEval

HumOmni 2026 · Track 2 (ProactivEval) — proactive streaming video QA

Watch a video at 2 fps, given one question at t=0, and decide when to speak and what to say — causally (using only frames with timestamp ≤ t). Ranking score = 0.9·PAUC + 0.1·(1−duplicate); hidden judge = Gemini 3.1 Flash-Lite. Baseline MMDuet2 = 53.3; public leader at the time = 60.21.

This repository is the Phase-1 system: a causal streaming orchestrator around a hosted VLM (qwen/qwen3.7-plus via OpenRouter). No training and no local large models — it runs on a single 8 GB GPU.

The method in one paragraph

The hidden judge is recall-only — it asks whether the accumulated response text covers the reference answer's key points, with no precision penalty for extra detail. So the winning strategy is comprehensive coverage: fire on every frame (gate off), prompt the VLM to describe the scene exhaustively and read all on-screen text verbatim, and strip only true repeats with an LLM-entailment dedup. The final config queries a small prompt ensemble per frame — v5 (exhaustive + OCR) + v3 (comprehensive) + v8 (few-shot, teach-by-example) — and merges their drafts. The big lever was the prompt, not timing or a bigger model.

Result (WEB dev set, faithful Gemini judge, untouched holdout gold[400:500])

Caveat: WEB-dev is the leakage-free dev set, not the phase1 test set (which has no ground truth). The v8 gain is partly holdout-overfit — the robust gain over v5+v3 is ~+0.002–0.005. Our PAUC harness is verified bit-exact against the official scorer, and the grader is confirmed Gemini 3.1 Flash-Lite.

Project layout

src/humomni/
  core/     streaming_driver · context · pauc_eval · judges · emb_cache · env_util · validate_submission
  phase1/   vlm_client · dedup · policy_api · policies · perception_gate
  tuning/   cache · faithful_eval
scripts/    build_submission.py · run_inference.py · infer_simple.py
tests/      test_causality.py · test_inference_parity.py
config.json   .env.example   pyproject.toml

Run every command from the repo root — modules use repo-root-relative data/ and cache/ paths.

Setup

python -m venv .venv && .venv\Scripts\activate        # Windows (use: source .venv/bin/activate on *nix)
pip install -e .                                       # installs humomni.* + deps (pyproject.toml)
cp .env.example .env                                   # then fill in the keys (below)

.env needs OPENROUTER_API_KEY (the VLM) and VERTEX_KEY (the Gemini judge used for the LLM-entailment dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set at data/phase1/data/<id>/ — frames 0.5.jpg, 1.0.jpg, … (.png also supported, as shipped with the phase-2 test set) plus question.json per sample; --data_dir selects the test set (per-prompt caches are kept separate per phase automatically).

Build the submission — one command

config.json holds the entire method ("ensemble": ["v5","v3","v8"] @ 40, dedup/theta, and the perception "gate" disabled by default — use_gate:false gives the shipped max-score point; enabling it at novelty ≥ 16 / silence 3 s selects a lower-cost operating point):

# phase-1 test set
python scripts/build_submission.py --data_dir data/phase1/data --out submissions/tryanderror2.jsonl
# phase-2 test set — same pipeline; its predictions use the EXH+COMP @ k32 operating point
# (config_phase2.json), selected for cost; see Table I of the report
python scripts/build_submission.py --data_dir data/phase2/data --config config_phase2.json --out submissions/tryanderror2_phase2.jsonl

--data_dir points at any folder of <id>/ sample dirs (frames 0.5.jpg/0.5.png, 1.0.…, plus question.json; both extensions are auto-detected). For each ensemble prompt it ensures a per-prompt VLM decision cache — building it from the test frames if missing (paid: ~1 VLM call per prompt per frame; resumable, skips cached frames) — then replays the merge + LLM-entailment dedup and validates the output (all ids present, times multiples of 0.5). Re-runs are free once the caches exist. Runs on different test sets never mix: decision/novelty caches are namespaced per data folder (cache/phase1_*.jsonl vs cache/phase2_*.jsonl), and the resume file is fingerprinted by (data path + config) so a leftover from a different run is set aside instead of resumed.

The live equivalent produces byte-identical output (paid on every run — use when caches are absent):

python scripts/run_inference.py --policy api --workers 8 --out submissions/tryanderror2.jsonl
python scripts/infer_simple.py  --policy api             --out submissions/tryanderror2.jsonl   # minimal twin

Both drive the same APIPolicy from config.json, so all three commands point at the same method (live ≡ replay). Upload is manual: the file must be named exactly your Team ID.

Tests

python tests/test_causality.py          # causality assert fires on shuffled / out-of-order frames
python tests/test_inference_parity.py   # run_inference == infer_simple; ensemble multi-emit per frame

Compliance

Track 2 permits any open-source or commercial LLM API (organizer clarification, 2026-07-07), with one condition: when generating a response at time t, only frames at time ≤ t may be used as visual input. This system enforces that structurally (below) and tests it.

The one hard rule — causality

Acting at time t uses only frames with timestamp ≤ t. Enforced structurally in humomni.core.streaming_driver.drive (assert t > last_t) and guarded by tests/test_causality.py (the shuffled-frame test). Keep it green always.