Turkish-LLaVA-v0.1-Q4_K_M-GGUF

Turkish-LLaVA-v0.1-Q4_K_M-GGUF

This model is a converted and quantized version of ytu-ce-cosmos/Turkish-LLaVA-v0.1 vision-language model using llama.cpp.

Usage

You can use the model with llama-cpp-python package as following:

from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llama3VisionAlphaChatHandler

llm = Llama(
  model_path="Turkish-LLaVA-v0.1-Q4_K_M.gguf", # path to language model
  n_gpu_layers=-1, # for running on GPU
  chat_handler=Llama3VisionAlphaChatHandler(
      # path to image encoder
      clip_model_path="Turkish-LLaVA-v0.1-mmproj-F16.gguf",
  ),
  seed=1337, # for reproducing same results
  n_ctx=4096, # n_ctx should be increased to accommodate the image embedding
  verbose=False, # disable the logging
)

# url for the input image
url = "https://huggingface.co/ytu-ce-cosmos/Turkish-LLaVA-v0.1/resolve/main/example.jpg"

messages = [
    {"role": "system", "content": "Sen yardımsever bir asistansın."},
    {
        "role": "user",
        "content": [
            {"type" : "text", "text": "Bu resimde neler görüyorsun?"},
            {"type": "image_url", "image_url": {"url": url}}
        ]
    },
]

response = llm.create_chat_completion(
    messages=messages,
    max_tokens=64,
)

print(response["choices"][0]["message"]["content"])
# Output: Resimde, sarı çiçeklerle çevrili bir köpek yavrusu görülüyor.