AgriScope

AgriScope

<p align="center"> <strong>Pixel-Grounded Multimodal Understanding for Agriculture Images</strong> </p> <p align="center"> <strong>Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain, Sajid Javed</strong><br> Khalifa University of Science and Technology, Abu Dhabi, UAE </p> <p align="center"> <a href="https://github.com/boudiafA/AgriScope"><img src="https://img.shields.io/badge/Project%20repository-GitHub-181717" alt="AgriScope project repository on GitHub"></a> <img src="https://img.shields.io/badge/Paper-Coming%20soon-lightgrey" alt="Paper coming soon"> <img src="https://img.shields.io/badge/Weights-Coming%20soon-lightgrey" alt="Weights coming soon"> <a href="annotations/README.md"><img src="https://img.shields.io/badge/AgriGround-Annotations%20available-success" alt="AgriGround annotations available"></a> </p>

Release status: AgriGround train/test annotations, task definitions, project figures, and reported manuscript results are available here. The AgriScope project repository is available on GitHub. Model weights, configuration files, processors, and model training/inference code are coming soon.

Model Description

<p align="center"> <img src="images/overview.png" alt="AgriScope and AgriGround overview" width="100%"> </p>

AgriScope is a pixel-grounded multimodal model for agricultural image understanding. It supports image-level, region-level, and pixel-level interaction within one framework, connecting generated agricultural concepts to segmentation masks and spatial annotations.

The model is designed to identify, describe, localize, and segment agricultural entities including plant diseases, lesions, pests, crops, weeds, botanical species, plant organs, and structural features. It supports both text-only responses and responses grounded in masks or normalized bounding boxes.

AgriScope is trained with AgriGround, a large-scale pixel-grounded agricultural instruction-tuning dataset containing 503,919 images and 11,421,148 samples across 14 tasks.

Model Details

Architecture

<p align="center"> <img src="images/architecture.png" alt="AgriScope architecture" width="100%"> </p>

AgriScope combines complementary semantic and spatial pathways:

1. A biological contextual encoder extracts agricultural and biological semantics.
2. A global contextual encoder preserves texture, morphology, boundaries, and dense spatial information.
3. Projection layers map visual representations into the multimodal language space.
4. The language model generates text and special [SEG] tokens for grounded concepts.
5. Each [SEG] hidden state conditions the SAM2-driven mask decoder to produce a corresponding pixel-level mask.

The manuscript implementation initializes its visual and grounding components from BioCLIP, DINOv3, and SAM2. The large pretrained encoders remain frozen while alignment modules and LoRA parameters are optimized.

Inputs

• An agricultural RGB image
• A natural-language instruction or question
• An optional region for region-conditioned tasks
• Optional conversation history for multi-turn interaction

Outputs

• Natural-language captions and answers
• Interleaved grounded captions with phrase-mask correspondence
• Referring-expression, semantic, and part segmentation masks
• Region-conditioned descriptions and conversations
• Counts and normalized bounding boxes
• Multi-turn grounded responses

Intended Uses

Primary Research Uses

• Agricultural image captioning and visual question answering
• Plant disease, pest, weed, crop, and species understanding
• Referring expression and semantic segmentation
• Grounded caption generation
• Region-level and multi-turn grounded interaction
• Agricultural object counting, detection, and localization
• Research on interpretable and evidence-grounded agricultural AI

Out-of-Scope Uses

• Autonomous pesticide, treatment, or crop-management decisions without expert review
• Safety-critical agricultural robotics without independent perception safeguards
• Regulatory, insurance, or legal determinations
• Identification of entities outside the supported visual and agricultural domains

AgriGround Training Data

<p align="center"> <img src="images/annotation_pipeline.png" alt="AgriGround annotation pipeline" width="100%"> </p>

AgriGround is produced using a four-stage annotation and task-generation pipeline:

1. Generate detailed image captions, class descriptions, counts, and bounding-box metadata.
2. Correct captions, identify grounded object phrases, and prepare segmentation prompts.
3. Generate and align segmentation masks with grounded phrases.
4. Synthesize instruction-following records for the 14 supported tasks.

Dataset Statistics

Supported Tasks

<p align="center"> <img src="images/task_examples_v3.png" alt="Task-specific AgriGround examples" width="62%"> </p>

See docs/TASKS.md for definitions and per-task sample counts.

The train/test annotations are available in annotations/, with file-level checksums and counts in annotations/manifest.json. Source agricultural images are not redistributed; image_path identifies the corresponding image in its source dataset, whose original license and terms remain applicable.

Training Procedure

Training follows two stages described in the manuscript:

1. Visual-language and grounding alignment: optimize projection layers, grounding modules, and the segmentation decoder while keeping the pretrained backbones frozen.
2. Instruction tuning: apply parameter-efficient LoRA adaptation using the 14 AgriGround tasks while retaining frozen visual encoders.

The joint objective combines autoregressive language modeling with binary cross-entropy and Dice losses for segmentation supervision. Full hyperparameters, preprocessing, and reproducibility scripts will be released with the code.

Evaluation

The following results are reported in the current manuscript draft.

Efficiency

Complete baseline comparisons, cross-dataset evaluation, ablations, and experimental settings will accompany the paper release.

Qualitative Results

Grounded Caption Generation

<p align="center"> <img src="images/qualitative_gcg.png" alt="AgriScope grounded caption generation examples" width="100%"> </p>

Representative Tasks

<p align="center"> <img src="images/qualitative_tasks.png" alt="AgriScope representative task results" width="100%"> </p>

Referring Expression Segmentation

<p align="center"> <img src="images/referring_segmentation_comparison.png" alt="Referring expression segmentation comparison" width="100%"> </p>

Limitations and Risks

• AgriScope can produce plausible but incorrect descriptions, classifications, counts, or masks.
• Performance may degrade under poor illumination, blur, occlusion, unusual viewpoints, severe domain shift, or very small targets.
• Fine-grained diseases, species, and pests with similar visual characteristics may be confused.
• Segmentation quality depends on the visual coverage and annotation quality of the training data.
• Dataset composition may encode geographic, crop, acquisition, and class-frequency biases from its source datasets.
• Outputs require review by qualified agricultural experts before being used for diagnosis or management decisions.

Repository Contents

.
|-- README.md
|-- CITATION.cff
|-- LICENSE
|-- annotations/
|   |-- README.md
|   |-- manifest.json
|   |-- validation.json
|   |-- train/
|   `-- test/
|-- docs/
|   `-- TASKS.md
`-- images/
    |-- overview.png
    |-- architecture.png
    |-- annotation_pipeline.png
    |-- task_examples_v3.png
    |-- qualitative_gcg.png
    |-- qualitative_tasks.png
    `-- referring_segmentation_comparison.png

Release Roadmap

Development updates and future code releases are tracked in the AgriScope GitHub repository.

Citation

The final paper link and citation will be added upon release. Until then, please use:

@misc{boudiaf2026agriscope,
  title  = {AgriScope: Pixel-Grounded Multimodal Understanding for Agriculture Images},
  author = {Boudiaf, Abderrahmene and Alanssari, Mohamad and Hussain, Irfan and Javed, Sajid},
  year   = {2026},
  note   = {Manuscript under review}
}

License

Repository documentation and code are provided under the Apache License 2.0. The source images are not redistributed and remain subject to their original dataset licenses and terms.

Acknowledgments

This work was conducted at Khalifa University of Science and Technology, Abu Dhabi, UAE. We acknowledge the creators and maintainers of the agricultural datasets and open-source foundation models that support this research.

Contact

For questions and collaborations, use the AgriScope GitHub issue tracker or the Hugging Face Community tab.