prefailurenet-ui-prefail-v2

RESEARCH BENCHMARK, NOT FOR DEPLOYMENT. This package is a synthetic counterfactual benchmark for studying ordered-temporal evidence in pre-failure detection. It is not a browser-agent guardrail and has no real-world validation.

PreFailureNet / UI-PreFail-v2

PreFailureNet / UI-PreFail-v2 is a synthetic ordered temporal counterfactual benchmark for pre-failure detection in browser-agent-like workflows. It includes shortcut probes, baseline ladders, a reference Transformer, and calibrated gate diagnostics. It is not a production browser-agent guardrail.

Headline Result vs Shallow Baseline

Sources: transformer_v2_35e_panel.json, baselines.json (entry all_frames).

A shallow logistic regression over flattened frames already reaches AP 0.8762 on this split, so the v2 headline reflects a benchmark that does not require deep temporal modeling. See v3 below for a split where this gap closes.

v3: xor_order_color defeats v2

v3 introduces the xor_order_color family (local commit 3a323b1, file src/prefailurenet/data/synthetic.py, function _temporal_order_v3_pair). The label is the XOR of an event-order bit and a per-pair color bit painted identically into every frame, so ordered-frame linear probes lose their handle while the pair structure stays intact.

On v3 (test seed 2920, n_test=256, 128/128 pairs), the all-frames ordered LR drops from AP 0.8762 on v2 to AP 0.5486 on v3 (rounded to 0.549 in docs/v3_split_evaluation.md), which is below the project's 0.60 hard gate and within ~0.05 of chance. Source: ~/Projects/PreFailureNet/runs/benchmark_v3/baselines_seed2920/baselines.json (entry all_frames, field ap_failure_vs_stable), cross-referenced in docs/v3_split_evaluation.md.

The v2 Transformer recipe retrained on v3 flatlines:

Source: ~/Projects/PreFailureNet/runs/benchmark_v3/transformer_v3/metrics.json (raw values 0.5068114175400058 and 0.500152587890625).

v3 inverts the v2 narrative. v2 results should be read as a defeated benchmark, not as evidence that the v2 Transformer architecture learns temporal order.

v3 will be published as a separate HF repo or addendum after a v3-targeted model exists. Currently no model beats v3 baseline.

Exact Positioning

This package is a benchmark and research artifact. The correct claim is:

Ordered temporal evidence is required under synthetic counterfactual controls (v2). v3 shows this benchmark is itself defeated by an XOR-color counterfactual; the v2 headline is therefore a benchmark, not a capability claim.

Do not read this release as evidence of production browser-agent safety, real-world website generalization, or a deployable intervention gate.

What This Package Contains

• A public claims ledger.
• A compact artifact index.
• Benchmark v2 summary metrics.
• Shortcut probes and baseline ladder.
• Pair-matching and temporal-signal diagnostics.
• Reference Transformer v2 held-out test metrics.
• Risk-only gate calibration diagnostics.
• Reproduction commands.
• SHA256 checksums for all shipped artifacts (CHECKSUMS.sha256).

Raw NPZ arrays and model checkpoint weights are not included in this compact HF package. They are reproducible from the commands below.

Tested Environment And Disk Requirements

This milestone was validated on Ubuntu 22.04 / Linux 6.8, x86_64, Python 3.10.12. Use Python 3.10 for reproduction; the project metadata allows Python >=3.10, but the release validation environment is Python 3.10.

CUDA is optional for the published benchmark checks. The generator, diagnostics, shortcut baselines, HF package checker, and tests run on CPU. Transformer training can use CPU or CUDA depending on the local Torch install; no CUDA, Jetson, TensorRT, browser-extension, or live agent deployment result is claimed.

Disk planning:

• Compact HF package: under 1 MB.
• Reproduced v2 NPZ splits: about 40 MB for train/calibration/test.
• Existing runs/benchmark_v2 artifact directory: about 50 MB.
• Python environment with Torch, OpenCV, scikit-learn, and Pillow: budget 2-5 GB.

Why v1 Was Flawed

The original temporal counterfactual split was useful but flawed:

• It matched state, context, action tokens, and final frame.
• The label signal lived in 1-2 mid-sequence visual patches.
• Patch position and active steps were family-structured.
• A logistic regression on mid-frame pixels reached AP 0.9249 and pair-rank 0.9875.
• The strongest v1 Transformer reached AP 0.8746, so it still lost to the mid-frame LR probe.

Conclusion: v1 was shortcut-controlled, but it rewarded localized mid-frame patch detection more than temporal reasoning. v1 remains documented as a negative result and shortcut-control scaffold.

What v2 Fixes

UI-PreFail-v2 / temporal_order_v2 changes the label rule:

• Safe and unsafe samples have the same UI template/style/distractors.
• Safe and unsafe samples have identical state, context, and action tokens.
• Safe and unsafe samples have the same final frame.
• Safe and unsafe samples have the same unordered frame bag.
• The label depends on the order of two event-bearing frames.

This makes static, final-frame, non-vision, single mid-frame, and unordered-bag probes weak by construction. Ordered frame sequences and temporal deltas still contain useful signal. (v3 closes the remaining handle. See above.)

Held-Out v2 Pair Controls

Source: data_diagnostics.json.

Shortcut Probe Table

Source: baselines.json.

Reference Results

Source: transformer_v2_35e_panel.json and transformer_v2_minrecall080_fpr020.json.

Gate calibration selected on the calibration split with min_recall=0.80, max_false_intervention_rate=0.20:

This is a synthetic benchmark calibration result, not a deployment threshold.

Reproduction Commands

Run from the repository root:

PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role train \
  --num-samples 512 --seed 910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role calibration \
  --num-samples 256 --seed 1910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role test \
  --num-samples 256 --seed 2910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/temporal_data_diagnostics.py \
  --train runs/benchmark_v2/data/software_synthetic_n512_seed910.npz \
  --test runs/benchmark_v2/data/software_synthetic_n256_seed2910.npz \
  --output-dir runs/benchmark_v2/diagnostics

PYTHONPATH=src python3 scripts/temporal_baselines.py \
  --train runs/benchmark_v2/data/software_synthetic_n512_seed910.npz \
  --test runs/benchmark_v2/data/software_synthetic_n256_seed2910.npz \
  --output-dir runs/benchmark_v2/baselines

PYTHONPATH=src python3 scripts/train.py --config configs/benchmark_v2_transformer.yaml

PYTHONPATH=src python3 scripts/temporal_model_eval.py \
  --config configs/benchmark_v2_transformer.yaml \
  --checkpoint runs/benchmark_v2/transformer_v2/best.pt \
  --name transformer_v2_35e \
  --split test \
  --output-dir runs/benchmark_v2/model_panels

PYTHONPATH=src python3 scripts/calibrate_gate.py \
  --config configs/benchmark_v2_transformer.yaml \
  --checkpoint runs/benchmark_v2/transformer_v2/best.pt \
  --output runs/benchmark_v2/gate_calibration/transformer_v2_minrecall080_fpr020.json \
  --min-recall 0.80 \
  --max-false-intervention-rate 0.20

PYTHONPATH=src python3 scripts/build_benchmark_v2_summary.py \
  --root runs/benchmark_v2 \
  --output runs/benchmark_v2/benchmark_v2_summary.json

Package validation:

PYTHONPATH=src python3 scripts/check_hf_package.py

Artifact Map

Checksums

Verify with sha256sum -c CHECKSUMS.sha256 from the repo root.

Verified

• v2 generator exists.
• v2 matched ordered-event pairs exist.
• v2 shortcut probes exist.
• v2 baseline ladder exists.
• Transformer v2 result exists.
• Gate calibration result exists.
• v1 flaw is documented.
• v3 XOR counterfactual split exists locally and defeats both the ordered-LR baseline and the v2 Transformer recipe (see v3 section above).

Unverified / Not Claimed

• No production browser traffic.
• No live browser-agent deployment.
• No deployable browser-agent gate.
• No real-world website generalization.
• No claim that deep temporal reasoning is required.
• No claim that this benchmark proves browser-agent safety.

Limitations

• The benchmark is synthetic.
• The UI events are rendered primitives, not production browser traces.
• The reference result is on generated data only.
• A flattened all-frames LR is strong because it sees ordered frames; v2 tests ordered temporal evidence, not uniquely deep temporal reasoning. The v3 XOR split removes even this handle and currently has no model that beats baseline.
• Gate calibration is a benchmark diagnostic, not a safety policy.
• The class schema contains unused classes in this split.

License

Apache-2.0 for the package code, artifacts, and model card. Training and test data are synthetic and generated by the reproduction commands in this repo. No third-party dataset license applies.

Next Research Steps

• Add external browser-agent traces with consent and clear provenance.
• Add stronger adversarial temporal controls where full-frame linear probes degrade (v3 is the first such step).
• Add held-out UI event grammars and templates.
• Evaluate under real browser screenshots and DOM/action logs.
• Separate benchmark scoring from intervention-policy deployment work.
• Train a v3-targeted model that actually beats v3 baseline before publishing a v3 HF repo.