rf-detr-nano-bookshelf

RF-DETR-nano — Bookshelf / book detector

A lightweight, real-time detector that finds individual books (spines) on shelves. It is an RF-DETR-nano model finetuned in two stages: first on a large generic bookshelf dataset, then adapted to real library shelves. It is built as the detection stage of a shelf-monitoring pipeline (empty-space and linear-metre measurement are derived from the boxes downstream — see Downstream pipeline).

• Task: object detection (bounding boxes), single class book
• Base model: RF-DETR-nano (Roboflow, COCO-pretrained, Apache-2.0)
• Framework: rfdetr==1.9.4 · input resolution: 384 · license: Apache-2.0

Versions

Both training stages are frozen as git tags, so either is retrievable by revision (the files at main are the latest, stage-2 weights):

from huggingface_hub import hf_hub_download
ckpt = hf_hub_download("Geraldine/rf-detr-nano-bookshelf",
                       "checkpoints/checkpoint_best_ema.pth", revision="v2")  # or "v1"

How to use

from rfdetr import RFDETRNano
from PIL import Image
import supervision as sv

model = RFDETRNano(pretrain_weights=ckpt)
model.optimize_for_inference()

image = Image.open("shelf.jpg")
det = model.predict(image, threshold=0.5)          # supervision.Detections
annotated = sv.BoxAnnotator().annotate(image.copy(), det)
annotated.save("shelf_annotated.jpg")

For a wide shot (a whole range in one photo), run tiled inference — spines shrink below the model's scale otherwise. For CPU / on-premise deployment, see the ONNX section below.

Model comparison (compare_models.py)

scripts/compare_models.py compares the three detectors side by side on one image (or a folder) and writes a *_compare.jpg with three panels — base COCO / v1 Roboflow / v2 library — each with its boxes drawn and its book count in the title bar, plus a one-line terminal recap that states the tiling mode used:

shelf.jpg  [tiled=True]  base (COCO)=0 | v1 Roboflow=25 | v2 library=195  -> compare/shelf_compare.jpg

Two details that make the comparison honest:

• base = RFDETRNano() with no weights — the original COCO-pretrained model. Since COCO has a book class, its detections are filtered to that class (resolved from the model's class_names) so all three columns count the same thing.
• Tiling is decided once per image and applied to all three models (--tiled auto default): the script probes the frame with v2 and switches to tiled inference only when books are tiny or barely found full-frame (spines < 2 % of the image width, or 0 detections on a large frame). --tiled off / --tiled on force either mode; --no-base drops the COCO column.

pip install -U rfdetr huggingface_hub pillow numpy     # GPU recommended for 3 models + tiling
python scripts/compare_models.py --image shelf.jpg
python scripts/compare_models.py --images ./photos --tiled on
python scripts/compare_models.py --image wide.jpg --no-base   # v1 vs v2 only

v1 and v2 default to this repo's checkpoints loaded by tag (repo::checkpoints/checkpoint_best_ema.pth@v1 / @v2); pass --v1 / --v2 to compare other checkpoints. Example outputs on real library stacks live in compare_samples/.

Video / camera inference (video_infer.py)

scripts/video_infer.py runs the detector on a webcam or a video file. Shelf monitoring does not need a live 30 fps stream: for an aisle walk-through, the sensible mode is frame sampling, and that is what --stride provides.

pip install -U rfdetr supervision opencv-python pillow numpy huggingface_hub

python scripts/video_infer.py --source 0                        # webcam, live window, Esc to quit
python scripts/video_infer.py --source parcours.mp4 --out sortie.mp4 --stride 5   # file, 1 frame in 5
python scripts/video_infer.py --source parcours.mp4 --out sortie.mp4 --tiled      # wide/corridor shot

What the flags do:

• --source 0 — webcam (any OpenCV camera index). Live window with per-frame book count overlaid; Esc to quit. Needs a display; the file mode below runs headless.
• --stride N (file mode) — process 1 frame in N: the sensible sampling for a walk-through of the stacks. The output fps is rescaled accordingly, so the annotated video still plays back at real-time speed.
• --tiled (+ --tile 640, overlap 0.25) — tiled inference for wide or corridor footage where spines shrink below the model's scale.
• --revision v2 (default) — selects the checkpoint by tag (v1, v2, or any commit hash), exactly as in the Versions table — easy A/B of the two adaptation stages on your own footage.
• --threshold 0.5 (default), --out sortie.mp4 for the annotated output; file mode logs the per-frame book count every 20 frames.

CPU deployment with ONNX (no PyTorch)

For on-premise, GPU-free deployment, run the exported ONNX with onnxruntime — typically 2–4× faster than PyTorch on CPU, with no torch / rfdetr runtime dependency (only numpy, onnxruntime and pillow).

A ready-to-use export is hosted in this repo: onnx/rfdetr-nano-bookshelf.onnx — exported from the current (v2 / library-adapted) weights, frozen at the 384 px training resolution. The v1-domain export remains retrievable under the v1 tag.

Regenerating the export (e.g. after a new fine-tune):

pip install -U onnx          # up-to-date exporter — model.export() can fail on an old onnx
pip install "rfdetr[onnx]"   # onnx export extras

from rfdetr import RFDETRNano
model = RFDETRNano(pretrain_weights="checkpoints/checkpoint_best_ema.pth")
model.export()   # writes an .onnx into ./output — input resolution frozen at 384

Inference (numpy + onnxruntime only):

import numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

onnx_path = hf_hub_download("Geraldine/rf-detr-nano-bookshelf", "onnx/rfdetr-nano-bookshelf.onnx")
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

RES  = 384                                       # must match the exported resolution
MEAN = np.array([0.485, 0.456, 0.406], np.float32)
STD  = np.array([0.229, 0.224, 0.225], np.float32)

def preprocess(path):
    img  = Image.open(path).convert("RGB")
    W, H = img.size
    x = np.asarray(img.resize((RES, RES), Image.BILINEAR), np.float32) / 255.0
    x = (x - MEAN) / STD
    return x.transpose(2, 0, 1)[None], (W, H)    # NCHW float32, original size

x, (W, H) = preprocess("shelf.jpg")

# Match outputs BY NAME — RF-DETR does not guarantee positional order
out    = {o.name: v for o, v in zip(sess.get_outputs(), sess.run(None, {"input": x}))}
boxes  = out["dets"][0]                           # [num_queries, 4] cxcywh, normalized [0,1]
logits = out["labels"][0]                         # [num_queries, num_classes+1] raw logits

conf = (1.0 / (1.0 + np.exp(-logits))).max(1)     # sigmoid; single effective class -> max activation
keep = conf > 0.5
boxes, conf = boxes[keep], conf[keep]

# cxcywh (normalized) -> xyxy (pixels in the original image)
cx, cy, w, h = boxes.T
xyxy = np.stack([(cx - w/2)*W, (cy - h/2)*H, (cx + w/2)*W, (cy + h/2)*H], axis=1)
# (xyxy, conf) is what the downstream gap / linear-metre geometry stage consumes

Resolution: is the 384 resize still necessary? Yes. The export froze the input at 384 × 384 (the resolution the weights were trained at), and the pre/post-processing above depends on it. A different resolution requires re-exporting with RFDETRNano(resolution=...) — and even then, 384 remains the reference: the stage-2 fine-tune ran at 384, and accuracy is not guaranteed at other sizes.

Two things that silently break ONNX inference: using a resolution other than the 384 the model was exported at (boxes come out shifted/scaled), and reading dets / labels by position instead of by name. For a multi-class model, use argmax over the class dimension and map indices to names; here everything is book, so the max activation is enough.

CPU performance note. "Real-time" figures for RF-DETR are measured on GPU. On CPU, expect a few frames per second even in the nano size — fine for frame-sampled shelf surveys, not smooth live video.

Training

Stage 1 — generic bookshelf domain. Finetuned on Roboflow Universe bookshelf-recognition-2 (~5.9k images, single class book), 10 epochs at resolution 384, batch 16, lr 1e-4, EMA. → mAP@50 0.974 / mAP@50:95 0.802 (validation, human labels).

Stage 2 — real-library adaptation. Continued from the stage-1 checkpoint and adapted to real library shelves by distillation (pseudo-labeling) from GPT-6 Astra: 285 photos from llabres/library-dataset auto-labelled by the teacher (boxes requested on a 0–1000 normalized grid, cross-checked with Grounding DINO and visually triaged). Training mix: domain images ×4 oversampled + 800 replay images from stage 1 (to avoid forgetting), 40 epochs, seed 42, lr 1e-4, EMA (1,772 images / 79,555 boxes total). → mAP@50 0.6755 / mAP@50:95 0.4301 (best EMA, epoch 36).

Stage 2 is a distillation (pseudo-labeling) from GPT-6 Astra: training and validation boxes are teacher-generated. The mAP is therefore measured against teacher labels, not an independent human benchmark — and for the same reason it is not directly comparable to the stage-1 number (human labels, different domain).

checkpoints/: checkpoint_best_ema.pth (recommended) + _best_regular.pth + _best_total.pth. metrics.csv holds all curves; per-run history is in the repo's git log.

Downstream pipeline

This detector emits book boxes; a separate geometry stage turns them into empty spans, linear metres and per-shelf fill rates (row clustering, gap measurement, calibration). Empty space is derived geometrically, not detected as a class.

Limitations

• Domain coverage. Stage 2 adapts to a single real-library source; other furniture, lighting and angles may still need more representative images.
• Label ceiling. Stage-2 labels are VLM-generated; the model's accuracy is bounded by the auto-labeller's quality on this domain.
• Single class. Books only — empty gaps and shelf-end labels are handled downstream / not by this model. Low-resolution or very wide shots need tiled inference.

Citation & acknowledgements

• RF-DETR — Roboflow, https://github.com/roboflow/rf-detr
• Stage-1 data — bookshelf-recognition-2, Roboflow Universe
• Stage-2 images — llabres/library-dataset (MIT)
• Stage-2 auto-labels — GPT-6 Astra

@software{rf_detr_nano_bookshelf,
  author = {Geoffroy, Géraldine},
  title  = {RF-DETR-nano Bookshelf Detector (two-stage, library-adapted)},
  year   = {2026},
  url    = {https://huggingface.co/Geraldine/rf-detr-nano-bookshelf}
}