vernier-rung1-probe

vernier rung-1 probe

Not an instrument yet. A logistic-regression probe on frozen facebook/dinov2-small features, trained to reproduce gemini-2.5-flash's hand-count judgements on egocentric factory frames, wrapped in an abstention cascade calibrated on human gold. It does not clear its pre-registered bar, and this card exists so the failure is inspectable rather than absent.

floor_reached = false, holds = false. The cascade's own message: target floor 0.8 is unreachable (at 95% confidence) at any coverage > 0 on the given held-out gold

What it is

• Backbone: facebook/dinov2-small -- a disclosed substitute for the pre-registered facebook/dinov3-vits16-pretrain-lvd1689m, which is gated for this account (docs/DECISIONS.md D051/D061 in the repo).
• Features: mean-pooled patch tokens from the last hidden state.
• Preprocessing, reproduced by hand so torchvision is not a dependency: resize shortest edge to 256 (bicubic), center-crop 224×224, scale to [0, 1], normalise by ImageNet mean/std.
• Head: sklearn.linear_model.LogisticRegression, trained on n = 600 frames labelled by Build AI's own stored gemini-2.5-flash P0b output (the judge is the target on purpose: an instrument that improved on the thing it measures would stop measuring it).
• Cascade: abstain when predict_proba max is below a threshold chosen so the Wilson 95% lower bound on accuracy against held-out human gold clears the floor. On this data no threshold does.

Load

from vernier.distil.linear_probe import LinearProbe  # pip install git+https://github.com/caiotheodoro/vernier
probe = LinearProbe.load("rung1_probe.joblib")

Feature extraction (_preprocess, _extract_features) lives in the repo's scripts/distill_rung1.py; the weights alone are not the instrument.

Why publish a model that fails

The dataset release and the measurement card claim this artifact exists and reports these numbers. A loadable probe with a stated failure is checkable; a missing one is not. Human gold here is n = 93 (calibration 46, evaluation 47); a larger gold set, the pre-registered backbone, or the rung-2 LoRA could move this either way. None of that has been run.

Data: https://huggingface.co/datasets/caiotheodoro/vernier. Code and decision log: https://github.com/caiotheodoro/vernier. Why this exists and why it failed: https://caio.theodoro.dev/blog/vernier-judge-errors-run-one-way.

License

Apache-2.0.