moondream-prompt

Fine tuned version of moondream2 for prompt generation from images. Moondream is a small vision language model designed to run efficiently on edge devices. Check out the GitHub repository for details, or try it out on the Hugging Face Space!

Usage

pip install transformers timm einops bitsandbytes accelerate

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from PIL import Image

DEVICE = "cuda"
DTYPE = torch.float32 if DEVICE == "cpu" else torch.float16 # CPU doesn't support float16
revision = "ac6c8fc0ba757c6c4d7d541fdd0e63618457350c"
tokenizer = AutoTokenizer.from_pretrained("gokaygokay/moondream-prompt", revision=revision)
moondream = AutoModelForCausalLM.from_pretrained("gokaygokay/moondream-prompt",trust_remote_code=True,
    torch_dtype=DTYPE, device_map={"": DEVICE}, revision=revision)
moondream.eval()

image_path = "<image_path>"
image = Image.open(image_path).convert("RGB")
md_answer = moondream.answer_question(
        moondream.encode_image(image),
        "Describe this image and its style in a very detailed manner",
        tokenizer=tokenizer,
    )

print(md_answer)

Example

"a very angry old man with white hair and a mustache, in the style of a Pixar movie, hyperrealistic, white background, 8k"

<!-- gokaygokay-citation -->

Citation and attribution

This model release is maintained by Gökay Aydoğan. If you reference this repository in academic work, please cite it as follows and also cite the upstream models, datasets, or projects it builds upon.

@software{aydogan2024moondream_prompt,
  author = {Aydoğan, Gökay},
  title = {{moondream-prompt}},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/gokaygokay/moondream-prompt},
  note = {Model repository; cite the base model and upstream datasets as required.}
}