vit-historical-page

Image classification using fine-tuned ViT - for historical :bowtie: documents sorting

Goal: solve a task of archive page images sorting (for their further content-based processing)

Scope: Processing of images, training and evaluation of ViT model, input file/directory processing, class 🏷️ (category) results of top N predictions output, predictions summarizing into a tabular format, HF 😊 hub support for the model

Versions 🏁

There are currently 2 version of the model available for download, both of them have the same set of categories, but different data annotations. The latest approved v2.1 is considered to be default and can be found in the main branch of HF 😊 hub ¹ 🔗

Model description 📇

🔲 Fine-tuned model repository: vit-historical-page ¹ 🔗

🔳 Base model repository: Google's vit-base-patch16-224, vit-base-patch16-384, vit-large-patch16-284 ² [^13] [^14] 🔗

Data 📜

Training set of the model: 8950 images for v2.0

Training set of the model: 10745 images for v2.1

Training set of the model: 14565 images for v2.2, v3.2 and v5.2

Categories 🏷️

Evaluation set: 1290 images (taken from v2.2 annotations)

Data preprocessing

During training the following transforms were applied randomly with a 50% chance:

• transforms.ColorJitter(brightness 0.5)
• transforms.ColorJitter(contrast 0.5)
• transforms.ColorJitter(saturation 0.5)
• transforms.ColorJitter(hue 0.5)
• transforms.Lambda(lambda img: ImageEnhance.Sharpness(img).enhance(random.uniform(0.5, 1.5)))
• transforms.Lambda(lambda img: img.filter(ImageFilter.GaussianBlur(radius=random.uniform(0, 2))))

Training Hyperparameters

• eval_strategy "epoch"
• save_strategy "epoch"
• learning_rate 5e-5
• per_device_train_batch_size 8
• per_device_eval_batch_size 8
• num_train_epochs 3
• warmup_ratio 0.1
• logging_steps 10
• load_best_model_at_end True
• metric_for_best_model "accuracy"

Results 📊

v2.0 Evaluation set's accuracy (Top-3): 95.58%

v2.1 Evaluation set's accuracy (Top-3): 99.84%

v2.2 Evaluation set's accuracy (Top-3): 100.00%

v2.0 Evaluation set's accuracy (Top-1): 84.96%

v2.1 Evaluation set's accuracy (Top-1): 96.36%

v2.2 Evaluation set's accuracy (Top-1): 99.61%

Result tables

• v2.0 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v2.0 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-1_EVAL.csv 🔗

• v2.1 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v2.1 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-1_EVAL.csv 🔗

• v2.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v2.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-1_EVAL.csv 🔗

Table columns

• FILE - name of the file
• PAGE - number of the page
• CLASS-N - label of the category 🏷️, guess TOP-N
• SCORE-N - score of the category 🏷️, guess TOP-N
• TRUE - actual label of the category 🏷️

Contacts 📧

For support write to 📧 [email protected] 📧

Official repository: UFAL ³

Acknowledgements 🙏

• Developed by UFAL ⁴ 👥
• Funded by ATRIUM ⁵ 💰
• Shared by ATRIUM ⁵ & UFAL ⁴
• Model type: fine-tuned ViT with a 224x224 ² 🔗 or 384x384 [^13] [^14] 🔗 resolution size

©️ 2022 UFAL & ATRIUM

Footnotes

1. https://huggingface.co/k4tel/vit-historical-page ↩ ↩²

2. https://huggingface.co/google/vit-base-patch16-224 ↩ ↩²

3. https://github.com/ufal/atrium-page-classification ↩

4. https://ufal.mff.cuni.cz/home-page ↩ ↩²

5. https://atrium-research.eu/ ↩ ↩²