Florence-2-DaViT-large-ft

Disclaimer

This belongs to Microsoft Florence 2, all I have done is taken the weights and modified the code to be compatiable with Huggingface pretrained models. The reason why is I want to use Florence 2 and it's components with the Hugginface framework and ONNX.

DaViT Model

This repository contains the implementation of the DaViT (Dual-Attention Vision Transformer) model for image classification tasks. The model leverages dual attention mechanisms to improve performance on various image datasets.

Model Description

DaViT (Dual-Attention Vision Transformer) is designed to handle image classification tasks effectively. It combines spatial and channel attention mechanisms to capture intricate details in images. The model has multiple stages, each with convolutional embeddings and attention blocks.

Example

Here is an example of how to use the DaViT model for image classification:

# Load model directly
from transformers import AutoModel, AutoProcessor
from PIL import Image
import requests

model = AutoModel.from_pretrained("amaye15/DaViT-Florence-2-large-ft", trust_remote_code=True, cache_dir=os.getcwd())
processor = AutoProcessor.from_pretrained("amaye15/DaViT-Florence-2-large-ft", trust_remote_code=True, cache_dir=os.getcwd())


prompt = "<OCR>"
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt")

model(inputs["pixel_values"])

Credits

This model is inspired by and builds upon the ideas presented in the Florence-2-large model by Microsoft.