Pheye-x2-448

miguelcarv/Pheye-x2-448

Pheye-x2-448 is a image-text-to-text model from miguelcarv. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers.

- These models train a fraction of the number of parameters other models of similar sizes train - Are more efficient in how they incorporate vision into language tasks (dense cross-attention LLaVA style architecture)…

Downloads · 30 days

5% of all-time downloads

All-time downloads

245

Public

Repo size

11.7 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pt6.4 GB · 100%

At a glance

Task: Image-Text-to-Text
Library: transformers
Access: Public
Created: May 7, 2024
Updated: May 29, 2024
SHA: 49471c8f

Try a prompt

Model card

From the Hugging Face model README

Φ Pheye - a family of efficient small vision-language models

These models train a fraction of the number of parameters other models of similar sizes train
Are more efficient in how they incorporate vision into language tasks (dense cross-attention > LLaVA style architecture)
Are more efficient in how the process high resolution input images
Use less data to train yet achieve competitive results (if you want to redo this recipe using more data results will be even better)

Benchmarks

Model	Resolution	Trainable Params	Data	VQAv2	Nocaps	TextVQA	TextCaps
MoE-LLaVA-1.6B×4	336	2.9B	5.9M	76.0	-	47.8	-
MoE-LLaVA-2.7B×4	336	5.3B	5.9M	77.1	-	50.2	-
moondream1	384	1.86B	3.9M	74.7	-	35.6
moondream2	384	1.86B	-	77.7	92.5	49.7	120.2
Pheye-x4 🤗	448	295M	2.9M	75.2	110.3	45.9	106.4
Pheye-x4 🤗	672	295M	2.9M	75.5	110.8	49.2	111.9
Pheye-x2 🤗	448	578M	2.9M	76.0	111.8	47.3	108.9
Pheye-x2 🤗	672	578M	2.9M	76.4	110.5	50.5	115.9

Examples

Image	Example
<img src="https://c5.staticflickr.com/6/5463/17191308944_ae0b20bb7e_o.jpg" width="500"/>	How much do these popcorn packets weight?<br>These popcorn packets weigh 90g each.<br><br>What brand are these popcorn packets?<br>These popcorn packets are from the brand "Magic Pop".
<img src="https://farm2.staticflickr.com/2708/5836100440_6e1117d36f_o.jpg" width="500"/>	Can I pet that dog?<br>No, you cannot pet the dog in the image. The dog is a black bear, which is a wild animal and should not be approached or interacted with by humans. It is important to maintain a safe distance from wild animals to avoid any potential harm to both the animal and the person.
<img src="https://farm4.staticflickr.com/3898/15152219465_4fb6d266ff_o.jpg" width="500"/>	Why is the person wearing protective gear?<br>The person in the image is wearing protective gear, such as a beekeeper's suit and a helmet, because they are engaged in beekeeping. Beekeeping involves handling bees and their hives, which can be potentially dangerous due to the risk of bee stings. The protective gear helps to minimize the risk of bee stings and ensures the beekeeper's safety while working with bees.

Usage

To generate a sample response from a prompt use generate.py. Use a Python version >= 3.11. Start by cloning the repo and create a virtual environment with the necessary packages:

git clone https://github.com/miguelscarv/pheye.git
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Then run generate.py:

python3 generate.py --image_path images/dog_flower.jpg --prompt "What is the dog holding in it's mouth?" --device cuda

Acknowledgments

This implementation was inspired by OpenFlamingo's repository.

Task: Image-Text-to-Text
Library: transformers
Languages: en
Created: May 7, 2024
Updated: May 29, 2024