coco-comparator-bank

COCO Comparator Bank

One coordinate comparison per COCO category on EUPE-ViT-B features. A 768 pixel image gives 2304 patch tokens at the final layer; layernorm across the 768 channels and max-pool across patches gives one 768-D vector. Each category is decided by comparing two coordinates of that vector.

person   ⟺  feat[48]  > feat[637]
giraffe  ⟺  feat[436] > feat[713]
toilet   ⟺  feat[105] > feat[240]

from infer import CategoryBank
bank = CategoryBank.load()
present = bank.predict('image.jpg')      # {'person': True, 'chair': False, ...}

Results

Pairs are selected on COCO train2017, 118,287 images, and scored on val2017. Categories range from 54 % of images to under 0.2 %, so each is scored on a class-balanced subsample of val2017 and reported by balanced accuracy, where 0.500 is chance for every category regardless of frequency.

Median 0.8265, minimum 0.6702, maximum 0.9653. Per-category figures and the selected pair for each are in eval.json.

giraffe          0.9653      bench            0.6702
toilet           0.9430      backpack         0.6732
snowboard        0.9388      chair            0.6931
baseball glove   0.9350      handbag          0.7140
elephant         0.9326      cell phone       0.7150

Toaster and hair drier appear in 8 and 9 validation images, too few to support a balanced subsample, and are recorded in the skipped field of rules.json.

Shared reference axes

The negative side of the comparison repeats across categories. feat[240] is the reference for nine of them, including cat, chair, bed, toilet, refrigerator and book; feat[548] for eight, including cup, tv, laptop, keyboard, cell phone and toothbrush; feat[338] for fork, knife, spoon and cake. Ninety-five distinct channels carry all 78 rules.

feat[48] is the positive term for person, backpack and handbag, and the negative term for bird. It rises on people and on objects people carry, and falls on non-human animals.

Selection

choose.py searches exhaustively over ordered pairs drawn from the 128 dimensions with the largest class-mean separation for that category, scoring feat[i] > feat[j] by balanced accuracy on a class-balanced subsample of train2017 capped at 40,000 images. The search returns two indices. Nothing is fitted, and the 128-dimension restriction makes every figure a lower bound.

Dimension indices and the direction of each comparison are fixed structure. Each rule has no free parameters and two fixed ones.

Circuit

rtl_gen.py emits the bank as one Verilog module and synth.py synthesizes it with nosis for a Lattice ECP5 LFE5U-25F.

Inputs are the 95 channels as signed INT8, post-layernorm and post-max-pool. Outputs are 78 bits, one per category. Combinational, no multipliers, no memory and no constants, at 3.3 slices per category. Category names that collide with Verilog keywords take a trailing underscore, which is how fork reaches the port list.

tests/test_rtl.py checks every comparator against rules.json and simulates the bank against a Python reference under Icarus Verilog.

Layout

common/      pooled features, the comparison rule, metrics, named pools
cache.py     pooled feature cache for a COCO split
choose.py    pair selection on train2017, writes rules.json
verify.py    scoring on val2017, writes eval.json
rtl_gen.py   Verilog generation from rules.json
synth.py     nosis synthesis, writes synth.json
infer.py     loader for the bank
rtl/         the generated bank
tests/       consistency suite, no backbone or dataset required

Each measured JSON opens with a provenance block naming its generating script and the pool it read. tests/test_artifacts.py enforces the pairing and that selection and scoring name different splits.

Running

pip install -e .
python cache.py --split train2017
python cache.py --split val2017
python choose.py
python verify.py
make synth
make test

COCO_ROOT is the dataset root. BACKBONE is the backbone repo id or a local path. BACKBONE_SRC supplies argus.py from a local directory; otherwise it is fetched from the backbone repo. Caching the two splits is a backbone forward over 123,287 images; every later step reads the cache and takes seconds.

bfloat16 kernels select reduction orders by batch size, so cached values depend on --batch. A cache must be built at one batch size throughout.

Source backbone

EUPE-ViT-B from Meta FAIR (arXiv:2603.22387, Zhu et al., March 2026), distilled from PEcore-G + PElang-G + DINOv3-H+ via a 1.9B proxy teacher. License: FAIR Research License, non-commercial. These rules are artifacts derived from that backbone's feature geometry.