co-dino-5scale-lsj-vit-l-lvis

Co-DINO (ViT-L (304M, EVA-02), Objects365 pretrain, LVIS fine-tune (LSJ), LVIS)

<!-- Demo banner: 1x2 montage of tools/inference.py video outputs produced with THIS checkpoint. Media lives under assets/ in this repo. The <video> renders on the Hugging Face model page; on GitHub the nested <img> poster shows. --> <p align="center"><video controls autoplay loop muted playsinline width="900" poster="https://huggingface.co/dronefreak/co-dino-5scale-lsj-vit-l-lvis/resolve/main/assets/demo_banner_poster.jpg" src="https://huggingface.co/dronefreak/co-dino-5scale-lsj-vit-l-lvis/resolve/main/assets/demo_banner.mp4"><img src="https://huggingface.co/dronefreak/co-dino-5scale-lsj-vit-l-lvis/resolve/main/assets/demo_banner_poster.jpg" alt="Co-DINO ViT-L (304M, EVA-02) detections on two dashcam clips" width="900"></video></p> <p align="center"> <a href="https://huggingface.co/tasks/object-detection"><img src="https://img.shields.io/badge/Task-Object_Detection-blue?style=flat-square" alt="Task"></a> <a href="https://github.com/open-mmlab/mmdetection/tree/2.x"><img src="https://img.shields.io/badge/Framework-MMDetection_2.x-0aa1a7?style=flat-square" alt="Framework"></a> <a href="https://github.com/Sense-X/Co-DETR"><img src="https://img.shields.io/badge/Architecture-Co--DETR_(DINO)-purple?style=flat-square" alt="Architecture"></a> <a href="https://arxiv.org/abs/2211.12860"><img src="https://img.shields.io/badge/Venue-ICCV_2023-orange?style=flat-square" alt="Venue"></a> <br> <a href="#evaluation-as-reported-by-the-original-authors"><img src="https://img.shields.io/badge/LVIS_val_box_AP-68.0_(reported)-9cf?style=flat-square" alt="box AP"></a> <a href="#measured-inference-footprint"><img src="https://img.shields.io/badge/Params_(inference)-350.2M-lightgrey?style=flat-square" alt="Params"></a> <a href="co_dino_5scale_lsj_vit_large_lvis.py"><img src="https://img.shields.io/badge/Schedule-Objects365_pretrain,_LVIS_fine-tune_(LSJ)-lightgrey?style=flat-square" alt="Schedule"></a> <br> <a href="#license-status"><img src="https://img.shields.io/badge/Weights_license-undetermined-lightgrey?style=flat-square" alt="Weights license"></a> <a href="https://arxiv.org/abs/2211.12860"><img src="https://img.shields.io/badge/Paper-arXiv:2211.12860-B31B1B?style=flat-square" alt="Paper"></a> <a href="https://github.com/dronefreak/Co-DETR"><img src="https://img.shields.io/badge/Code-dronefreak/Co--DETR-black?style=flat-square" alt="Code"></a> <a href="#license-status"><img src="https://img.shields.io/badge/Objects365-research--only_pretrain-critical?style=flat-square" alt="Objects365 pretrain"></a> </p>

Disclaimer

This is not an official release. All credit for the method, the architecture, and the trained weights belongs entirely to the original authors: Zhuofan Zong, Guanglu Song, and Yu Liu (SenseTime X-Lab). I claim no contribution to the underlying research or training. I host the same checkpoint the authors published to Google Drive, with a model card and a copy of its config file.

Mirror rationale

This checkpoint is already on the authors' own Hub (zongzhuofan/co-detr-vit-large-lvis), so availability isn't the issue the way it is for the Google-Drive-only checkpoints elsewhere in this zoo. I mirror it anyway for consistency: the authors' repo ships a bare weights file with no config, so it isn't runnable on its own. This mirror bundles the matching config, a demo banner made with this checkpoint, and the same card format as every other checkpoint here.

I mirror only the pretrained weights; I do not redistribute any training or evaluation data. I will transfer or remove this mirror at the original authors' request. Please cite the original paper (see Citation) and use the official repository or the maintained fork dronefreak/Co-DETR for code.

---

Quickstart

This is not a plain-PyTorch / transformers model. Inference needs the Co-DETR project code and the OpenMMLab 1.x stack (MMDetection 2.25.3, MMCV-full 1.5.0, PyTorch 1.11). The maintained fork ships a one-command environment setup for exactly this.

# 1. Get the code + a validated environment
git clone https://github.com/dronefreak/Co-DETR && cd Co-DETR
bash tools/setup_codetr_env.sh          # builds the `codetr` conda env
conda activate codetr

# 2. Pull this checkpoint + its config from the Hub
pip install -U "huggingface_hub[cli]"
hf download dronefreak/co-dino-5scale-lsj-vit-l-lvis \
    co_dino_5scale_lsj_vit_large_lvis.pth co_dino_5scale_lsj_vit_large_lvis.py \
    --local-dir checkpoints/

# 3. Run detection on an image / folder / video / webcam
python tools/inference.py \
    --config checkpoints/co_dino_5scale_lsj_vit_large_lvis.py \
    --checkpoint checkpoints/co_dino_5scale_lsj_vit_large_lvis.pth \
    --input demo/demo.jpg --out-dir outputs/ --save-json

The bundled config is identical to projects/configs/co_dino_vit/co_dino_5scale_lsj_vit_large_lvis.py in the repo.

---

What is Co-DETR / this checkpoint

Co-DETR ("DETRs with Collaborative Hybrid Assignments Training") is a training scheme, not a new architecture. It attaches auxiliary heads that use one-to-many label assignment (an ATSS head and a Faster-R-CNN-style RoI head) alongside the DETR decoder's one-to-one Hungarian matching, and feeds the positive proposals those heads produce back into the decoder as extra queries. This makes the encoder's features more discriminative and speeds up convergence. The auxiliary heads are used only during training: this checkpoint's config sets eval_module='detr', so at inference only the Co-DINO query head runs and the RPN / RoI / ATSS heads add no inference-time computation to the deployed detector.

This checkpoint is Co-DINO (5 feature scales, large-scale jitter (LSJ)) with a ViT-L (304M, EVA-02) backbone:

• Paper: DETRs with Collaborative Hybrid Assignments Training, ICCV 2023
• Config: co_dino_5scale_lsj_vit_large_lvis.py (bundled; identical to the Co-DETR repo copy)

---

Intended use & limitations

Intended use. Research and engineering work that needs an object detector in the Co-DETR family trained on LVIS v1 val: as a baseline, a feature/detection backbone, or a starting point for fine-tuning on your own dataset.

Limitations.

• Trained and evaluated on LVIS v1; predicts 1203 long-tailed categories, not the 80 COCO classes. LVIS uses federated evaluation (not every category is exhaustively annotated in every image). Read the LVIS paper before comparing its AP to a COCO-80 number.
• Accuracy can degrade substantially on domain-shifted imagery (aerial, medical, document, non-natural images, etc.).
• The reported 68.0 box AP is the authors' number; I have not reproduced it (see Evaluation).
• No real-time claim. See measured latency for the actual timings I got.
• Runs only on the legacy OpenMMLab 1.x stack (MMDetection 2.25.3 / MMCV-full 1.5.0 / PyTorch 1.11); no transformers / pipeline() support, and inference: false on the Hub.
• The auxiliary training heads in the checkpoint are not exposed as inference outputs; only the DETR query head's detections are returned.

---

What I checked

Integrity and "does it load and run" checks only. This is not an accuracy reproduction:

• It loads in the codetr env (Python 3.8 / torch 1.11.0+cu113 / mmcv-full 1.5.0 / mmdet 2.25.3): not a full state-dict key match: 26 missing key(s) and unexpected keys: 2 *.fed_loss_weight (a training-only LVIS federated-loss buffer), none affecting inference.
• I ran it end-to-end through tools/inference.py on the two dashcam clips shown in the banner above (CPU); it produced valid annotated outputs.
• I did not re-measure LVIS v1 val AP; the 68.0 box AP below is the authors' number.

Measured inference footprint

Method. I built the model with mmdet.apis.init_detector, then ran inference_detector() on a single image, device='cpu', batch size 1, FP32: one warm-up call, then timed calls with time.perf_counter(). No GPU was used for this checkpoint: mine was occupied by another job at the time I mirrored it, so I fell back to CPU for both this table and the demo banner above (same code path, just --device cpu; see tools/inference.py). Peak memory is the process's peak resident set size (resource.getrusage(RUSAGE_SELF).ru_maxrss), a high-water mark since process start, not attributable to a single call the way the CUDA allocator's counters are for the GPU-measured cards in this zoo.

This model has not been timed on a GPU in this repository. Expect CPU latency in the tens of seconds per image for this backbone/decoder size; LVIS v1 val inference at scale needs a GPU.

---

Evaluation (as reported by the original authors)

Schedule: Objects365 pretrain, LVIS fine-tune (LSJ), 900 object queries, single-scale test, no TTA. I have not independently re-measured these numbers.

---

Training data

• Objects365 pretraining: ~2M images, 365 categories. Shao et al., Objects365: A Large-Scale, High-Quality Dataset for Object Detection, ICCV 2019 (paper). Objects365's terms restrict use to non-commercial research; see License status.
• LVIS v1 fine-tuning: 1203 categories on the COCO 2017 image set with different (federated, long-tailed) annotations. Gupta, Dollar & Girshick, LVIS: A Dataset for Large Vocabulary Instance Segmentation, CVPR 2019 (arXiv:1908.03195).
• Backbone initialization: ViT-L (304M, EVA-02) pretrained on EVA-02 MIM self-supervised pretrain, then Objects365 supervised pretrain.

I do not redistribute LVIS or COCO here. Get LVIS from lvisdataset.org (it reuses COCO 2017 images; see cocodataset.org for those) and review both datasets' own terms yourself. This checkpoint's backbone/detector was pretrained on Objects365 before the fine-tune above. Unlike the plain-COCO checkpoints in this zoo, Objects365's non-commercial-research terms are a real constraint here, not a hypothetical one.

---

License status

The Co-DETR source code is MIT-licensed (© 2022 SenseTime X-Lab, LICENSE). The upstream repository does not provide a separate, explicit license for this checkpoint's weights. The original authors have published other Co-DETR checkpoints (the ViT-L family) on Hugging Face with MIT metadata, but I do not treat that as definitive evidence that these weights are independently licensed under MIT.

Because the applicable rights are undetermined, I set the Hub metadata for this repo to license: unknown. Verify the applicable rights yourself before redistribution or commercial use.

⚠️ Objects365 pretrain. This checkpoint's backbone/detector was pretrained on the Objects365 dataset before being fine-tuned on LVIS. Objects365's own terms restrict it to non-commercial research use (Objects365 terms); that restriction plausibly carries through to weights derived from it. Treat this checkpoint as research-only unless you independently confirm otherwise; this is a stronger caveat than the general license: unknown above, not a substitute for it.

If you are one of the original authors and want this mirror removed or transferred, open an issue on this repo or contact dronefreak and I will action it.

---

Citation

@inproceedings{zong2023detrs,
  title={DETRs with Collaborative Hybrid Assignments Training},
  author={Zong, Zhuofan and Song, Guanglu and Liu, Yu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={6748--6758},
  year={2023}
}

The underlying detector and backbone:

@inproceedings{zhu2021deformable,
  title={Deformable DETR: Deformable Transformers for End-to-End Object Detection},
  author={Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2021}
}
@article{fang2023eva02,
  title={EVA-02: A Visual Representation for Neon Genesis},
  author={Fang, Yuxin and Sun, Quan and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue},
  journal={arXiv preprint arXiv:2303.11331},
  year={2023}
}
@inproceedings{shao2019objects365,
  title={Objects365: A Large-Scale, High-Quality Dataset for Object Detection},
  author={Shao, Shuai and Li, Zeming and Zhang, Tianyuan and Peng, Chao and Yu, Gang and Zhang, Xiangyu and Li, Jing and Sun, Jian},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2019}
}
@inproceedings{gupta2019lvis,
  title={LVIS: A Dataset for Large Vocabulary Instance Segmentation},
  author={Gupta, Agrim and Dollar, Piotr and Girshick, Ross},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019}
}

Built with MMDetection / MMCV (OpenMMLab).

---

Acknowledgements

I sincerely thank Zhuofan Zong, Guanglu Song, and Yu Liu for developing Co-DETR and releasing the pretrained weights, and the OpenMMLab team for MMDetection and MMCV. This mirror only makes one of their existing checkpoints easier and more reliable to obtain.

---

Provenance

co_dino_5scale_lsj_vit_large_lvis.pth, 1,421,831,793 bytes, SHA-256 6bdd296b925fbc7fc6fe05b21099e33aa1ecabb061d9b502aba4db840d6a010b. I copied it byte-for-byte and unmodified from the authors' own Hugging Face repo zongzhuofan/co-detr-vit-large-lvis.

Repo files: co_dino_5scale_lsj_vit_large_lvis.pth (weights), co_dino_5scale_lsj_vit_large_lvis.py (the MMDetection config needed to build the model), config.json (metadata summary; also what the Hub uses to count downloads), plus assets/demo_banner.mp4 / assets/demo_banner_poster.jpg for the card. config.json is descriptive only, inference: false, and is not a transformers config.