Downloads · 30 days
8
16% of all-time downloads
docsim-lora
gt-free-ocr-metrics/docsim-lora
docsim-lora is a image feature extraction model from gt-free-ocr-metrics. Use it for the image feature extraction task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A LoRA-adapted CLIP+DINOv2 similarity head trained to measure visual similarity between original document page scans and their OCR reconstructions. Released as part of the OmniDocBench Render-and-Compare project.
All-time downloads
51
Public
Repo size
—
Likes
2
Trending 2
Repo makeup
Click a slice to open those files.
.pt6.6 MB · 95%
At a glance
- Task
- Image Feature Extraction
- License
- apache-2.0
- Access
- Public
- Created
- May 2, 2026
- Updated
- May 2, 2026
- SHA
- 80a9358b
Base models
Model card
From the Hugging Face model README
DocSim LoRA
A LoRA-adapted CLIP+DINOv2 similarity head trained to measure visual similarity between original document page scans and their OCR reconstructions. Released as part of the OmniDocBench Render-and-Compare project.
Architecture
- Backbone A: CLIP ViT-B/32 (pretrained on LAION-2B via open_clip)
- Backbone B: DINOv2 ViT-B/14
- Projection head: two-layer MLP (hidden=512, output=256)
- LoRA adapters: rank 16, alpha 32, dropout 0.05 — applied to both CLIP and DINOv2
Training
Trained on 20,280 triplets (anchor original page, positive OCR reconstruction, negative mis-matched reconstruction) derived from the OmniDocBench Render-and-Compare dataset.
| Setting | Value |
|---|---|
| Epochs | 3 |
| Batch size | 16 |
| Learning rate | 1e-4 |
| Train triplets | 19,266 |
| Val triplets | 1,014 |
| Best val accuracy | 99.90% |
| Margin (triplet loss) | 0.1 |
Files
| File | Description |
|---|---|
lora_adapter_best/ | LoRA adapter at best validation accuracy |
lora_adapter_final/ | LoRA adapter at end of training |
head_state_best.pt | Projection head weights (best checkpoint) |
head_state_final.pt | Projection head weights (final epoch) |
config.json | Full architecture config |
Use lora_adapter_best/ + head_state_best.pt for inference.
Usage
Download via GT-free-ocr-metrics:
bash download_models.sh
Then run any DocSim-based method:
bash scripts/run_method.sh docsim_lora
License
Apache-2.0. The companion datasets (OmniDocBench Render-and-Compare) are CC-BY-NC-4.0.
- Task
- Image Feature Extraction
- License
- apache-2.0
- Languages
- en, zh
- Created
- May 2, 2026
- Updated
- May 2, 2026