broadleaf-weed-detector

Broadleaf Weed Detector (YOLO11 · single-class · Hailo-10H ready)

A single-class "broadleaf weed vs grass" object detector for a UGV (unmanned ground vehicle) spot-spraying robot. It answers the only question a sprayer needs: where are the broadleaf things that are not grass? — rather than trying to identify weed species. Grass is thin; broadleaf weeds (dandelion, dock, plantain, thistle, etc.) are broad — so a binary detector is both simpler and far more robust than species classification.

Trained by the llama-thunderdome vision platform.

TL;DR — which file do I use?

Single class: 0 = weed. Input 640×640. Run with tiled (SAHI-style) inference — see Deployment below; it's essential for small weeds.

Results

Held-out test split (in-domain)

Real-world footage (3 phone clips of a Wisconsin lawn, 425 frames)

It reliably boxes the dock / dandelion / plantain rosettes in the turf and does not box grass. (A prior model trained on iNaturalist single-plant photos scored ~0 on the same footage — wrong domain. This one works because the training data is dock-in-turf, the same domain as deployment.)

Deployment — use tiled inference

The weeds are small at a robot's camera height, so a single full-frame pass downscales them away. Split each frame into a 3×2 grid (≈640 px tiles, ~12% overlap), run the detector per tile, map boxes back to full-frame coordinates, and NMS-merge. This is what produced the 0.86 number.

# Reference tiled inference is in scripts/tiled_infer_eval.py (tiled_preds()).
from ultralytics import YOLO
model = YOLO("weights/broadleaf-yolo11s-640.pt")   # or load the .hef on Hailo
# conf ~0.30, single class "weed", host-side NMS

On Hailo-10H: load the .hef, feed 640×640 tiles, host-side NMS. Both HEFs passed T1 (parse-faithful) + T2 (int8-accuracy) validation; T3 (hardware simulation) is a warn pending sign-off on physical silicon (quant_report.json has the details).

Training data

Merged from public grass/weed detection datasets, all normalized to a single weed class and perceptually deduplicated (the core "grass-weeds" dock-in-turf set is re-hosted 6× across Roboflow/HF/Kaggle):

• 7,016 unique images / ~23,000 boxes, split 5,614 / 701 / 701.
• Core: the RF100 "grass-weeds" set = Rumex obtusifolius (broad-leaved dock) in grass, ground-level top-down — the deployment domain.
• Plus augmented-startups weeds (4,203), an aerial weed set, and a nettle/ thistle set for diversity.

Full provenance (every source, version, license, and how it was downloaded + normalized) is in dataset/SOURCES.md. The normalized labels are in dataset/labels.tar.gz, images in dataset/images.tar.gz, and dataset/data.yaml is the Ultralytics config.

Reproduce / make a new version

Everything needed to rebuild or extend is included:

1. scripts/merge_ext_train.py — re-downloads (via the source list), normalizes to single-class, perceptual-dedups, splits, and trains yolo11n/s @640. Edit the INCLUDE set to add/drop sources.
2. scripts/video_to_frames.py + scripts/tile_label_footage.py — turn your own field footage into tiled Gemini-labeled training data (the flywheel). Mixing a little in-domain footage into the training set is the recommended way to push detection past 0.9 on your specific camera/lawn.
3. scripts/tiled_infer_eval.py — evaluate any .pt on footage (full vs tiled inference).

training_meta.json records the exact training config (epochs, batch, imgsz, base weights, seed).

Files

weights/   broadleaf-yolo11{n,s}-640.pt + .onnx
hailo/     broadleaf-yolo11{n,s}-640-hailo10h.hef (+ .har, quant_report.json)
eval/      test_eval.json, results.csv, PR/F1 curves, footage demo overlays
dataset/   data.yaml, labels.tar.gz, images.tar.gz, SOURCES.md
scripts/   merge_ext_train.py, tiled_infer_eval.py, tile_label_footage.py, video_to_frames.py
training_meta.json

License & attribution

Model weights released under CC-BY-4.0. The training images come from several public datasets with their own licenses (see dataset/SOURCES.md) — respect those when redistributing the images. Weed species in the core set: Rumex obtusifolius (broad-leaved dock / "ridderzuring").