vt-detr

<div align="center">

🔍 VT-DETR — Textbook Layout Detection

RF-DETR Large fine-tuned for Vietnamese textbook document layout analysis

</div>

---

Overview

VT-DETR stands for Vietnamese Textbook Detection Transformer — a fine-tuned variant of RF-DETR Large targeting automated detection of structural elements in Vietnamese textbook pages. It identifies four layout classes with high accuracy, enabling downstream document parsing, indexing, and educational content extraction pipelines.

---

Classes

---

Demo

---

Benchmark

✅ Best validation metrics

📉 Last epoch metrics

Metrics sourced from outputs/rfdetr_textbook/metrics.csv (full training run). A separate single-image eval run in outputs/exp_rfdetr_textbook/eval/metrics.json (num_images=1) is not representative and excluded from this report.

---

Training Configuration

model:
  size: large

training:
  epochs: 20
  batch_size: 4
  grad_accum_steps: 8
  resolution: 640
  device: cuda

augmentation:
  mosaic: true
  mixup: 0.1

early_stopping:
  enabled: true
  monitor: mAP50:95
  patience_logs: 4
  min_delta: 0.005

Dataset split (COCO format)

data_resplit/
├── train/
│   └── _annotations.coco.json
├── valid/
│   └── _annotations.coco.json
└── test/
    └── _annotations.coco.json

---

Usage

Option 1 — HuggingFace Transformers (recommended)

pip install transformers rfdetr

from transformers import AutoModel, AutoImageProcessor
from PIL import Image

processor = AutoImageProcessor.from_pretrained(
    "hietraan/vt-detr",
    trust_remote_code=True
)
model = AutoModel.from_pretrained(
    "hietraan/vt-detr",
    trust_remote_code=True
)

image = Image.open("page.png")
inputs = processor.preprocess(image)
outputs = model(inputs["images"], threshold=0.5)
results = processor.post_process_object_detection(outputs, threshold=0.5)

# results[0]["boxes"]  → xyxy coordinates (tensor)
# results[0]["scores"] → confidence scores (tensor)
# results[0]["labels"] → class ids (tensor)
print(results)

Note: trust_remote_code=True is required because this model uses a custom modeling_vtdetr.py. The checkpoint (~134MB) is downloaded automatically to ~/.cache/huggingface/ on first run.

Option 2 — RF-DETR directly

pip install "rfdetr[train]"

from rfdetr import RFDETRLarge
from huggingface_hub import hf_hub_download
from PIL import Image

weights = hf_hub_download(repo_id="hieptran318204/vt-detr", filename="checkpoint_best_total.pth")
model = RFDETRLarge(pretrain_weights=weights)

image = Image.open("page.png")
detections = model.predict(image, threshold=0.5)
print(detections)

Train / Evaluate (local)

python -m rfdetr_finetune.train
python -m rfdetr_finetune.evaluate

---

Repository Structure

hieptran318204/vt-detr
├── README.md                    # Model card
├── config.json                  # HuggingFace model config
├── preprocessor_config.json     # Image processor config
├── modeling_vtdetr.py           # Custom modeling code (trust_remote_code)
├── checkpoint_best_total.pth    # Best checkpoint (by mAP50:95)
├── inference.webp               # Demo output
└── page077.png                  # Demo input

---

Limitations

• Trained and evaluated on Vietnamese textbook domain; generalization to other document types is not guaranteed.
• Test set benchmark not yet finalized — reported metrics are from the validation split.
• Robustness may be improved with additional data augmentation, larger datasets, or LR schedule tuning.

---

License

This model is released under the Apache License 2.0, inherited from the base RF-DETR Large model by Roboflow.

Copyright 2025 Roboflow, Inc.
Licensed under the Apache License, Version 2.0. See LICENSE for details.

Fine-tuned weights and additional code in this repository are also distributed under Apache 2.0.

---

Citation

If you use this model or the RF-DETR framework, please cite:

Base model (RF-DETR Large):

@misc{rf-detr,
  title        = {RF-DETR: Neural Architecture Search for Real-Time Detection Transformers},
  author       = {Isaac Robinson and Peter Robicheaux and Matvei Popov and Deva Ramanan and Neehar Peri},
  year         = {2025},
  eprint       = {2511.09554},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2511.09554}
}

Base weights: Roboflow/rf-detr-large — Apache 2.0

This fine-tuned model (VT-DETR):

@misc{vtdetr2025,
  title        = {VT-DETR: RF-DETR Large Fine-tuned for Textbook Layout Detection},
  author       = {hieptran318204},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/hieptran318204/vt-detr}}
}

---

<div align="center"> _{Fine-tuned with ❤️ for Vietnamese educational document processing} </div>