howc

HOWC : Hierarchical Open-World Classification for road perception

<p> <a href="https://arxiv.org/abs/2608.07577"><img src="https://img.shields.io/badge/arXiv-2608.07577-b31b1b.svg" alt="arXiv"></a> <a href="https://youtu.be/Awy3qmCCe78"><img src="https://img.shields.io/badge/YouTube-explainer-red.svg?logo=youtube" alt="Video"></a> <a href="https://github.com/freshNfunky/IE2025-Research-Paper"><img src="https://img.shields.io/badge/GitHub-source-181717.svg?logo=github" alt="GitHub"></a> <img src="https://img.shields.io/badge/license-CC%20BY--NC%204.0-lightgrey.svg" alt="License"> <img src="https://img.shields.io/badge/training-free-2fd6a6.svg" alt="Training-free"> </p>

A training-free layer that turns a flat object detector into a hierarchical, open-set one. Every detection is resolved by taxonomic abstraction: the most specific level the evidence safely supports, or an explicit UNKNOWN OBSTACLE, never a confident wrong leaf.

<video autoplay loop muted playsinline width="100%" poster="https://huggingface.co/freshNfunky/howc/resolve/main/assets/hero_poster.jpg"> <source src="https://huggingface.co/freshNfunky/howc/resolve/main/assets/hero.mp4" type="video/mp4"/> <img src="https://huggingface.co/freshNfunky/howc/resolve/main/assets/hero_poster.jpg" alt="HOWC live tool: a deer crossing a road is classified as Living Being, with the taxonomy descent path and a segmentation cross-check" width="100%"/> </video>

_{The live tool. A deer, a class no closed detector is asked to output on the road, is not dropped and not mislabeled: it resolves to <code>Living Being</code>, with the full taxonomy descent path and an independent segmentation cross-check shown alongside.}

Watch the 3-minute explainer

<p align="center"> <a href="https://youtu.be/Awy3qmCCe78"> <img src="https://img.youtube.com/vi/Awy3qmCCe78/maxresdefault.jpg" alt="HOWC explainer video" width="70%"> </a> </p>

---

The problem

A closed-set detector must assign every object one of a fixed set of labels. On an object outside that set (a horse-drawn carriage, road debris, livestock on a rural road) it can only do one of two unsafe things: force a confident but wrong specific label, or drop the object entirely. Real incidents trace back to exactly this failure.

How it works

HOWC puts a taxonomy over the detector and, per object, aggregates leaf probability mass up the tree. It commits only as deep as one branch holds enough mass; otherwise it stops at a coarser node. A per-branch safety floor (Vehicle, Living Being, Static Object) means it never collapses into a useless bare Object: below the floor, it emits an explicit UNKNOWN OBSTACLE with a fully inspectable decision path.

<img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/taxonomy_architecture.png" alt="The object taxonomy, with safety floors marked" width="100%">

How it decides (worked examples)

Each detection produces a decision path: probability mass per node, the safety floor, and the resulting action.

<table> <tr> <td width="50%" valign="top"> <img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/ex_road_anomaly_04_2_abstracted.png" alt="ABSTRACTED: a truck YOLO calls 'car' is abstracted up to the safe Vehicle floor" width="100%"> <br>_{<b>ABSTRACTED.</b> YOLO calls a distant truck "car" (0.33). The mass is ambiguous below the <code>Vehicle</code> floor, so HOWC does not guess a leaf: it reports <b>Vehicle</b>, still useful, never wrong.} </td> <td width="50%" valign="top"> <img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/ex_road_anomaly_02_1_unknown.png" alt="UNKNOWN: mass splits above the safety floor, so the object is flagged UNKNOWN OBSTACLE" width="100%"> <br>_{<b>UNKNOWN.</b> A rural-road anomaly YOLO calls "person" (0.72). The mass splits <i>above</i> the floor across Living Being / Vehicle / Static, so HOWC refuses a category and flags <b>UNKNOWN OBSTACLE</b>.} </td> </tr> </table>

Beyond 2D: a depth cue for flat detections

A 2D box cannot tell a real vehicle from a flat picture of one (a billboard, a truck livery). The open-world study adds a monocular-depth relief cue: detections that are geometrically flat get flagged, and the depth field feeds the open-world proposal front-end.

<img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/depth3d_04.png" alt="Left: 3D mode flags flat detections on a horse trailer scene. Right: monocular depth (nearer = bright)" width="100%">

_{Left: 3D mode flags flat detections. Right: the monocular depth field (nearer = bright). This is a feasibility signal, not a solved module, and the paper is explicit that no single 2D cue suffices.}

Results (honest)

Out-of-vocabulary objects (v3 ground-truth leave-classes-out on COCO, n=235). Hold seven classes out of the taxonomy, then classify their real annotated crops:

<img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/v3_openworld_benchmark.png" alt="Flat head is 100% confidently wrong; HOWC is 0% confidently wrong and 94% safe" width="90%">

On known, in-taxonomy objects: 0% off-branch (categorical) errors with ~24% calibrated abstention, versus a flat arg-max head's ~53% off-branch errors.

The safety/specificity trade-off is a dial, not a fixed point. Every hierarchical configuration stays 100% safe on novel objects while still usefully classifying up to 76% of known ones; the flat baseline sits alone at 0% novel-safe.

<img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/calibration_tradeoff.png" alt="Safety vs specificity: every hierarchical config is 100% novel-safe; the flat baseline is 0% novel-safe" width="80%">

What this is, and is not. This is a safety result, not a specificity one. On closed-set COCO mAP a trained YOLO is more accurate, and we say so. The contribution is the abstraction layer over open-vocabulary (CLIP) features: it never makes a confident categorical mistake on an unknown object, at the honest cost of 69% abstention. The point is the tail, not the leaderboard.

An independent second path

The box classification is cross-checked against a class-agnostic segmentation path: does the pixel evidence back up the box verdict? Across sampled detections it confirms or stays neutral 63% of the time and flags 37% for review, with 0% hard conflicts, and it agrees most where it should (on abstracted and unknown objects).

<img src="https://raw.githubusercontent.com/freshNfunky/IE2025-Research-Paper/main/figures/segmentation_agreement.png" alt="Segmentation cross-check verdict: 37% confirm, 26% neutral, 37% flag, 0% conflict" width="70%">

Run it locally

Self-contained: code + taxonomy + a Gradio app. Training-free (pretrained YOLO + CLIP zero-shot); first run downloads weights (~360 MB).

pip install -r requirements.txt
python app.py            # Gradio UI: upload an image, see the taxonomy decision path
python app.py --share    # same, but also prints a temporary public URL (~72h)

Papers

• Open-world (v3): Open-World Hierarchical Perception: Taxonomic Abstraction over Class-Agnostic Proposals for the Safe Handling of Out-of-Vocabulary Road Objects, F. Schaller, arXiv:2608.07577.
• Foundational (v1): Hierarchical Taxonomic Abstraction for the Safe Handling of Novel Objects in Autonomous Driving Perception, doi:10.5281/zenodo.21593472.
• Source & full history: https://github.com/freshNfunky/IE2025-Research-Paper

@misc{schaller2026howc,
  title         = {Open-World Hierarchical Perception: Taxonomic Abstraction over
                   Class-Agnostic Proposals for the Safe Handling of
                   Out-of-Vocabulary Road Objects},
  author        = {Schaller, Felix},
  year          = {2026},
  eprint        = {2608.07577},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.07577}
}

License

CC BY-NC 4.0, matching the paper.