gemma-4-4b-it-latex-ocr-lora

Gemma 4 (4B) - LaTeX OCR LoRA

Model Description

LoRA adapter fine-tuned on Gemma 4 (4B) for LaTeX OCR — converting math formula screenshots into LaTeX code.

Fine-tuned with RsLoRA + DoRA strategy on the unsloth/LaTeX_OCR dataset (68,686 samples).

Training Details

Performance

Evaluation on 100 random test samples:

Training Curves

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from peft import PeftModel

# Load base model + LoRA adapter
model = Gemma4ForConditionalGeneration.from_pretrained(
    "google/gemma-4-4b-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, "itsawaysguthing/gemma-4-4b-it-latex-ocr-lora")

processor = AutoProcessor.from_pretrained("google/gemma-4-4b-it")

# Inference
image = Image.open("math_formula.png")
text = f"<bos><|turn|>user\n<|image|>Please output the LaTeX code for this math formula.<turn|><|turn|>model\n"
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False, num_beams=1)
out = out[:, inputs["input_ids"].shape[1]:]
result = processor.tokenizer.decode(out[0], skip_special_tokens=True).strip()
print(result)