Florence-2-finetuned-HuggingFaceM4-DocumentVQA

Florence-2-finetuned-HuggingFaceM4-DOcumentVQA

This model is a fine-tuned version of microsoft/Florence-2-base-ft on HuggingFaceM4/DocumentVQA dataset.

It is the result of the post Fine tuning Florence-2

It achieves the following results on the evaluation set:

• Loss: 0.7168

Model description

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.

He has also been finetuned in the docVQA task.

Training and evaluation data

This is finetuned on HuggingFaceM4/DocumentVQA dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-6
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
• num_epochs: 3

Training results

Framework versions

• Transformers 4.43.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1