Finetuned-Llava

Fine-Tuned LLAVA Model

This repository hosts the fine-tuned LLAVA model files, which have been adapted for data parsing and extracting JSON information from image reciepts. The model was fine-tuned on cord-v2 dataset.

Model Details

Model Versions

• LLAVA 1.6 Mistral 7B
  Fine-tuned version on Cord-V2 datasets.

How to Use

You can load and use this model directly from the HuggingFace Hub with the transformers library. Below is an example of how to load the model:

from transformers import AutoProcessor, BitsAndBytesConfig, LlavaNextForConditionalGeneration

MODEL_ID = "llava-hf/llava-v1.6-mistral-7b-hf"
REPO_ID = "abhisheksinghrathore/Finetuned-Llava"

processor = AutoProcessor.from_pretrained(MODEL_ID)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16
)
model = LlavaNextForConditionalGeneration.from_pretrained(
    REPO_ID,
    torch_dtype=torch.float16,
    quantization_config=quantization_config,
)

image = Image.open(io.BytesIO(image_bytes))

# Prepare input
prompt = f"[INST] <image>\nExtract JSON [/INST]"
max_output_token = 256
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
output = model.generate(**inputs, max_new_tokens=max_output_token)
response = processor.decode(output[0], skip_special_tokens=True)

# Convert response to JSON
generated_json = token2json(response)

---

To see the fine-tuning process and training configurtaton please visit this GitHub repository.

Additional Resources

• GitHub Repository for Fine-Tuning LLAVA
• A link to a YouTube video will be added here soon to provide further insights and demonstrations.