SmolVLM256M_CropDisease

SmolVLM-256M — Nepali Crop Disease

A LoRA fine-tune of HuggingFaceTB/SmolVLM-256M-Instruct that answers "What crop is shown in this image, and does it have any disease?" for photos of Nepali crops. Trained on w4ashabii/nepali_crop_data, a ShareGPT-format visual-QA dataset covering 9 crops.

Model Details

Training Data

• Dataset: w4ashabii/nepali_crop_data
• Format: ShareGPT-style conversations (sharegpt.jsonl), one user turn (image + question) → one assistant turn (crop + disease answer)
• Train examples used: 39,987
• Eval examples used: 500 (held-out validation split)
• Image preprocessing: resized to longest_edge=512, image splitting disabled (not needed for single-leaf/crop photos)

Training Procedure

Checkpoints were saved locally every 500 steps and mirrored to this Hub repo roughly every hour during training via a custom TrainerCallback (each upload replaced the repo's prior contents), with a final clean upload of just the adapter + processor after training completed.

Training / Validation Loss

Final train loss: 0.2363 (mean over all logged steps, includes early-training values before it stabilized). Validation loss plateaued around 0.183 from step ~3500 onward, suggesting the model converged with room left in the schedule rather than overfitting.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

BASE_MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
ADAPTER_REPO = "w4ashabii/SmolVLM256M_CropDisease"

processor = AutoProcessor.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForVision2Seq.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).to("cuda")
model.eval()

image = Image.open("your_crop_photo.jpg").convert("RGB")
question = "What crop is shown in this image, and does it have any disease? If so, name the disease."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=64)

answer = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer.strip())

With Unsloth (faster inference load)

from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained("w4ashabii/SmolVLM256M_CropDisease")
FastVisionModel.for_inference(model)
# ... same generate() call as above

Intended Use

• Assistive identification of crop type and visible disease symptoms from photos, for agricultural extension, research, or educational tooling focused on Nepali-grown crops.
• Not intended as a sole basis for treatment or pesticide decisions — outputs should be verified by an agronomist or local agricultural extension service, especially for high-stakes crop management decisions.

Limitations

• Small base model (256M params) and LoRA-only tuning trade some accuracy for speed/size; expect a lighter-weight, less nuanced answer than larger VLMs.
• Loss computed over the full sequence (question + answer) rather than answer-only, so the model was also lightly trained to reproduce the (fixed) question text — this doesn't appear to have hurt convergence here but is a simplification versus masked/answer-only supervision.
• Coverage limited to the crops and disease classes present in w4ashabii/nepali_crop_data; performance on out-of-distribution crops, lighting conditions, or camera angles is untested.
• Not evaluated for robustness to adversarial or low-quality images (blur, occlusion, multiple crops in frame, etc.).

Training Framework

Fine-tuned with 🤗 transformers.Trainer and LoRA adapters loaded/attached via Unsloth's FastVisionModel (use_gradient_checkpointing="unsloth") for reduced VRAM use and faster steps on a single T4 GPU.