Downloads · 30 days
123
13% of all-time downloads
KushagraWadhwa/medical-prescription-ocr-india
medical-prescription-ocr-india is a image-text-to-text model from KushagraWadhwa. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
A vision-language model fine-tuned to extract structured information from handwritten and printed Indian medical prescriptions. Given a prescription image, it returns a JSON object containing doctor details, patient d…
Downloads · 30 days
123
13% of all-time downloads
All-time downloads
969
Public
Parameters
3.8B
7.5 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors7.5 GB · 100%
From the Hugging Face model README
A vision-language model fine-tuned to extract structured information from handwritten and printed Indian medical prescriptions. Given a prescription image, it returns a JSON object containing doctor details, patient details, medications (with dosage, frequency, duration), diagnosis, and notes.
Extract structured data from medical prescription images in healthcare applications, EMR (Electronic Medical Record) systems, or patient monitoring dashboards. Input a prescription image, receive a structured JSON output.
Always have a qualified healthcare professional verify the model's output before acting on it. Do not use this model as the sole source of truth for medication dispensing or clinical workflows.
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from PIL import Image
import torch
model_id = "KushagraWadhwa/medical-prescription-ocr-india"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
model_id,
device_map="auto",
torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(model_id)
SYSTEM_PROMPT = """You are a medical prescription OCR assistant.
Extract all information from the prescription image and return it as JSON.
Return ONLY valid JSON, no explanation text.
Use this exact schema:
{"doctor_name": "", "clinic_name": "", "patient_name": "", "patient_age": "",
"date": "", "medications": [{"drug_name": "", "dosage": "", "frequency": "",
"duration": "", "instructions": ""}], "diagnosis": "", "notes": ""}
If a field is not visible, use empty string."""
image = Image.open("prescription.jpg").convert("RGB")
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "image", "image": image},
{"type": "text", "text": "Extract all prescription information from this image and return as JSON."}
]}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generated = outputs[0][inputs["input_ids"].shape[1]:]
print(processor.decode(generated, skip_special_tokens=True))
Example output:
{
"doctor_name": "Dr. R. K. Sharma",
"clinic_name": "Sharma Clinic",
"patient_name": "Suresh Patel",
"patient_age": "52",
"date": "12/03/24",
"medications": [
{
"drug_name": "Tab. Metformin 500mg",
"dosage": "500mg",
"frequency": "1-0-1",
"duration": "30 days",
"instructions": "After food"
},
{
"drug_name": "Tab. Amlodipine 5mg",
"dosage": "5mg",
"frequency": "OD",
"duration": "30 days",
"instructions": "Morning"
}
],
"diagnosis": "T2DM, HTN",
"notes": "Review after 1 month. RBS fasting."
}
naazimsnh02/medocr-vision-dataset<s_ocr> tag format into structured JSON with fields: doctor_name, clinic_name, patient_name, patient_age, date, medications, diagnosis, notesdrug_name, dosage, frequency, duration, instructions| Parameter | Value |
|---|---|
| Training regime | bf16 mixed precision |
| Epochs | 3 |
| Per-device train batch size | 1 |
| Gradient accumulation steps | 8 (effective batch size: 8) |
| Learning rate | 2e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 0.05 |
| Max sequence length | 512 |
| Optimizer | AdamW |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| LoRA target modules | q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4 (bitsandbytes) with double quantization |
246 samples from the validation split of naazimsnh02/medocr-vision-dataset.
Performance varies by:
The model accurately extracts structured fields from both handwritten and printed Indian prescriptions. Fields like doctor_name, clinic_name, and date are extracted reliably. Medication list accuracy depends on handwriting legibility. Hindi/mixed-language content is supported but may have lower accuracy than English-only prescriptions.
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
transformerspefttrl (SFTTrainer / SFTConfig)bitsandbytesacceleratedatasetstorch (cu124)Kushagra Wadhwa