dhivehi-image-text-10k-multi-gemma

Dhivehi Image Text Extraction (Document) — Gemma 3 Vision-Language Model

This project fine-tunes google/gemma-3-4b-pt with LoRA and image-text instruction data to extract Dhivehi text from images. The model was trained on synthetic documents with multiple lines of text and is instruction-aligned to perform optical character recognition (OCR) for Dhivehi.

Model Details

• Base Model: google/gemma-3-4b-pt
• Training Method: Parameter-efficient finetuning with LoRA (Low-Rank Adaptation)
• Instruction Format:

  Instruction: "Extract the dhivehi text from the image"
  Input: <Image>
  Output: <Dhivehi Text>
  

• Data Type: OCR-style synthetic images containing sentence-level Dhivehi text
• Dataset: alakxender/dhivehi-sl-image-text-all

Training Configuration

Final Training Metrics

{
  "final_loss": 0.0085,
  "mean_token_accuracy": 0.9985,
  "train_loss_avg": 0.2938,
  "train_runtime": "54,560 seconds (15.2 hours)",
  "samples/sec": 0.916,
  "total_flops": "1.28e18",
  "train_steps": 12500,
  "grad_norm": 0.0471,
  "learning_rate": 1.617e-6
}

Evaluation Results (First 10 Samples, Test Set)

Most predictions exactly match the ground truth or differ only slightly due to OCR-style variance.

How to Use

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained("alakxender/dhivehi-image-text-10k-multi-gemma")
processor = AutoProcessor.from_pretrained("alakxender/dhivehi-image-text-10k-multi-gemma")

instruction = "Extract the dhivehi text from the image"
image = Image.open("example.png").convert("RGB")

conversation = [{
    "role": "user",
    "content": [
        {"type": "text", "text": instruction},
        {"type": "image", "image": image}
    ]
}]

inputs = processor.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
model_inputs = processor(text=inputs, images=[image], return_tensors="pt").to(model.device)

generated = model.generate(**model_inputs, max_new_tokens=128)
print(processor.tokenizer.decode(generated[0], skip_special_tokens=True))

Model Files

• adapter_config.json, adapter_model.safetensors – LoRA weights
• pytorch_model.bin (optional, if merged)
• preprocessor_config.json, tokenizer_config.json – tokenizer
• README.md, eval_results.md

Citation

If you use this model, please cite or link to this repo.