PairDETR

PairDETR: face_body_detection_and_association

This card contains the official weights of PairDETR, a method for Joint Detection and Association of Human Bodies and Faces CVPR 2024. <img src="./teaser.jpg" width="1024" height="600"></img>

To reproduce our training experiments and evaluation results please use our github repo <a href="https://github.com/mts-ai/pairdetr">PairDETR</a>

System architecture:

<img src="./sys.jpg" width="1024" height="600"></img> PairDETR extracts embeddings using ResNet-50 followed by a transformer to predict pairs. During training, pairs are matched with ground-truth and corrected using approximated matching loss.

Inference example with transformers:

import os
import numpy as np 
import pandas as pd 
from transformers import DeformableDetrForObjectDetection, DeformableDetrConfig, AutoImageProcessor
import torch.nn as nn
import torch
from PIL import Image
import shutil
import requests
from hf_utils import PairDetr, inverse_sigmoid, forward

## Or download the weights manually
def get_weights():
    url = "https://huggingface.co/MTSAIR/PairDETR/blob/main/pytorch_model.bin"
    response = requests.get(url, stream=True)
    with open('full_weights.pth', 'wb') as out_file:
        shutil.copyfileobj(response.raw, out_file)

## loading the model
configuration = DeformableDetrConfig("SenseTime/deformable-detr")
processor = AutoImageProcessor.from_pretrained("MTSAIR/PairDETR")
model = DeformableDetrForObjectDetection(configuration)
model = PairDetr(model, 1500, 3)
get_weights()
checkpoint = torch.load("full_weights.pth", map_location="cpu")
model.load_state_dict(checkpoint, strict=False)

## run inference
path = "./test.jpg"
image = Image.open(path)
inputs = processor(images=image, return_tensors="pt")
outputs = forward(model, inputs["pixel_values"])

Results

Comparison between PairDETR method and other methods in the miss Matching Rate mMr-2 (the lower the better) on CrowdHuman dataset:

References and useful links

Papers

• <a href='https://arxiv.org/abs/2005.12872'>End-to-End Object Detection with Transformers</a>
• <a href='https://arxiv.org/abs/1805.00123'>CrowdHuman: A Benchmark for Detecting Human in a Crowd</a>
• <a href='https://openaccess.thecvf.com/content/ICCV2021/html/Wan_Body-Face_Joint_Detection_via_Embedding_and_Head_Hook_ICCV_2021_paper.html'>Body-Face Joint Detection via Embedding and Head Hook</a>
• <a href='https://arxiv.org/abs/2010.04159'>Deformable DETR: Deformable Transformers for End-to-End Object Detection</a>
• <a href='https://arxiv.org/abs/2012.06785'>DETR for Crowd Pedestrian Detection</a>
• <a href='https://arxiv.org/abs/2204.07962'>An Extendable, Efficient and Effective Transformer-based Object Detector</a>

This work is implemented on top of:

• <a href='https://github.com/facebookresearch/detr/tree/3af9fa878e73b6894ce3596450a8d9b89d918ca9'>DETR</a>
• <a href='https://github.com/fundamentalvision/Deformable-DETR'>Deformable-DETR</a>
• <a href='https://github.com/AibeeDetect/BFJDet/tree/main'>BFJDet</a>
• <a href='https://huggingface.co/docs/transformers/en/index'>Hugginface transformers</a>