rt-detr-r18-thermal-detection

RT-DETR-R18 Thermal Detection

This model is an RT-DETR-R18 object detector fine-tuned from PekingU/rtdetr_r18vd on the HIT-UAV high-altitude infrared thermal dataset. It detects people and vehicles in thermal imagery captured from unmanned aerial vehicles (UAVs).

Model details

The model predicts five labels:

DontCare is retained as a model label because it was present in the training annotations. Applications may choose to filter this label from displayed predictions.

Intended uses

This model is intended for research, prototyping, and portfolio demonstrations involving axis-aligned object detection in aerial thermal/infrared imagery, especially scenes similar to HIT-UAV. It may also serve as a starting point for further fine-tuning on related, appropriately licensed datasets.

Out-of-scope uses

The model is not intended for autonomous safety-critical decisions, biometric identification, individual tracking, surveillance without appropriate legal and ethical review, or deployment where an incorrect detection could directly cause harm. It has not been validated for visible-light imagery, ground-level cameras, medical imaging, or environments substantially different from HIT-UAV.

Dataset

HIT-UAV contains 2,898 infrared thermal images extracted from UAV video in scenes such as roads, parking lots, schools, and playgrounds. The dataset covers different flight altitudes, camera angles, and daylight conditions and contains 24,899 annotated objects.

The split used for fine-tuning was:

The training label set was Bicycle, Car, DontCare, OtherVehicle, and Person. Per-class instance counts were not preserved in the local model artifacts, so a numerical class distribution is not reported here. See the dataset publication for the authoritative dataset statistics and annotation details.

Training

The model was fine-tuned for 5 epochs with Hugging Face Transformers and PyTorch. The saved training arguments record a per-device train/evaluation batch size of 32, an initial learning rate of 5e-5, cosine scheduling, weight decay of 1e-4, FP16 training, and seed 42. The final checkpoint is stored directly as model.safetensors.

No quantitative evaluation output was available among the exported artifacts. Consequently, this card does not claim mAP, precision, recall, or other evaluation metrics.

Inference

Install the packages from requirements.txt, then replace the placeholder model ID below with this repository's Hugging Face ID.

import torch
from PIL import Image
from transformers import RTDetrForObjectDetection, RTDetrImageProcessor

MODEL_ID = "whate/rt-detr-r18-thermal-detection"
IMAGE_PATH = "path/to/thermal_image.jpg"
THRESHOLD = 0.5

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

processor = RTDetrImageProcessor.from_pretrained(MODEL_ID)
model = RTDetrForObjectDetection.from_pretrained(MODEL_ID).to(device)
model.eval()

image = Image.open(IMAGE_PATH).convert("RGB")
inputs = processor(images=image, return_tensors="pt")
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    outputs = model(**inputs)

# target_sizes uses (height, width) in the original, unresized image.
target_sizes = torch.tensor([image.size[::-1]], device=device)
results = processor.post_process_object_detection(
    outputs,
    target_sizes=target_sizes,
    threshold=THRESHOLD,
)[0]

for score, label_id, box in zip(
    results["scores"], results["labels"], results["boxes"]
):
    label = model.config.id2label[int(label_id)]
    x_min, y_min, x_max, y_max = box.tolist()
    print(
        f"{label}: {float(score):.3f} "
        f"at [{x_min:.1f}, {y_min:.1f}, {x_max:.1f}, {y_max:.1f}]"
    )

The processor converts the image to three-channel RGB, rescales pixel values, and resizes it to 640 × 640 according to preprocessor_config.json. The post-processing step converts normalized predictions back to xyxy boxes in the original image dimensions.

Example results

The following qualitative comparisons show predictions from the fine-tuned model alongside the original pretrained RT-DETR-R18. Confidence thresholds are visible in the images. These examples are illustrative and are not a quantitative benchmark.

<details> <summary>Additional qualitative examples</summary>

</details>

Limitations

• Performance metrics are not available in the exported local artifacts.
• The model may inherit geographic, environmental, sensor, altitude, viewpoint, and class-distribution biases from HIT-UAV.
• Small, low-contrast, partially occluded, or densely grouped objects remain challenging in high-altitude thermal imagery.
• Predictions can change with the confidence threshold. Validate and calibrate the threshold for the intended environment.
• The DontCare label may not be meaningful for every downstream application.
• The model was trained for axis-aligned bounding boxes and does not output oriented boxes, segmentation masks, identities, or tracks.

Licenses

The base model is distributed under the Apache License 2.0. This model repository uses the same Apache-2.0 license designation. HIT-UAV is distributed under CC BY 4.0; use of the dataset requires appropriate attribution to its authors. Dataset content is not included in this model repository.

Users are responsible for confirming that their use complies with all applicable model, dataset, privacy, and local legal requirements.

Acknowledgements

Thanks to the HIT-UAV authors for collecting and publishing the dataset, the RT-DETR authors for the detector architecture, and the Hugging Face community for the Transformers implementation and pretrained checkpoint.

Citation

If you use the dataset, cite HIT-UAV:

@article{suo2023hit,
  title     = {HIT-UAV: A high-altitude infrared thermal dataset for Unmanned Aerial Vehicle-based object detection},
  author    = {Suo, Jiashun and Wang, Tianyi and Zhang, Xingzhou and Chen, Haiyang and Zhou, Wei and Shi, Weisong},
  journal   = {Scientific Data},
  volume    = {10},
  pages     = {227},
  year      = {2023},
  doi       = {10.1038/s41597-023-02066-6}
}

For RT-DETR, cite:

@inproceedings{zhao2024detrs,
  title     = {DETRs Beat YOLOs on Real-time Object Detection},
  author    = {Zhao, Yian and Lv, Wenyu and Xu, Shangliang and Wei, Jinman and Wang, Guanzhong and Dang, Qingqing and Liu, Yi and Chen, Jie},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2024}
}