ja-med-clef-model

Model Card for Model ID

LLaVA-LLaMA 3 8B Fine-tuned Adapter

This repository contains the LoRA adapter weights and processor files for fine-tuning of LLaVA LLaMA 3 8B on the ROCO_v2 dataset for dual tasks:

• Medical Image Concept Detection
• Medical Image Captioning

The adapter allows applying the fine-tuning over the existing base model for reproducibility and lightweight sharing of modifications.

tags:

• medical
• vision-language
• image-captioning
• concept-detection
• llava
• llama3
• peft
• lora

Model Description

This repository contains the LoRA fine-tuned adapter for LLaVA LLaMA 3 8B applied to our dataset.

The base model used is:

• xtuner/llava-llama-3-8b-v1_1-transformers

You must load this base model and apply this adapter to reproduce our results.

Usage example

from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel from huggingface_hub import snapshot_download

Download adapter + processor

adapter_path = snapshot_download(repo_id="JoVal26/ja-med-clef-model")

Load base model

model_base = AutoModelForCausalLM.from_pretrained("xtuner/llava-llama-3-8b-v1_1-transformers")

Load tokenizer

tokenizer = AutoTokenizer.from_pretrained(f"{adapter_path}/final_processor")

Apply LoRA adapter

model = PeftModel.from_pretrained(model_base, f"{adapter_path}/final_lora_adapter_explicit")

Intended Use

This model is designed for use in clinical vision-language research and evaluation challenges with the following subtasks:

Concept Detection Task

Identify presence and location of relevant clinical concepts from medical images.
It serves as a building block for scene understanding and supports downstream image retrieval and clinical decision support.

Evaluation metrics: precision, recall, F1 (set coverage metrics).

Caption Prediction Task

Generate coherent medical captions describing an image, focusing on the interplay of visible elements and detected concepts for full-scene interpretation.

Evaluation metrics: BLEU, CIDEr, METEOR, ROUGE, and clinical concept coverage.

Example inference

inputs = tokenizer("Describe the clinical findings in this radiology image:", return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

This model is for research purposes only. It has been fine-tuned on an extended version of ROCO_v2 and may not generalize to other medical imaging datasets or modalities. Clinical use without additional validation is NOT recommended.

Citation

This fine-tuned adapter repo: JoVal26/ja-med-clef-model

License

This work is released under CC BY-NC 4.0 license. For non-commercial research use only.

• Developed by: JoVal26
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: Visual
• Language(s) (NLP): English
• License: CC BY-NC 4.0
• Finetuned from model [optional]: xtuner/llava-llama-3-8b-v1_1-transformers

Model Sources [optional]

Fine-tuned model from xtuner/llava-llama-3-8b-v1_1-transformers.

• Repository: JoVal26/ja-med-clef-model
• Paper [optional]: https://www.dei.unipd.it/~faggioli/temp/clef2025/paper_179.pdf
• Demo: https://github.com/Jangulo7/med_explain_ja

Usage and License Restrictions

• This adapter is shared for research and academic use via the Hugging Face Hub only.
• Downloading, redistribution, or commercial use is NOT permitted.
• Please use the model via the Hugging Face inference endpoints or via transformers' direct loading (no local weight download).
• For commercial licensing or dataset/model access, please contact the author.

License: CC BY-NC 4.0 (non-commercial use only)

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

How to Cite

If you use this adapter in academic work or publications, please cite:

BibTeX

@misc{JoVal26-ja-med-clef-model,
  title = {LLaVA-LLaMA 3 8B Fine-tuned Adapter},
  author = {JoVal26},
  year = {2025},
  howpublished = {\url{https://huggingface.co/JoVal26/ja-med-clef-model}},
  note = {Fine-tuned adapter for LLaVA-LLaMA 3 8B for medical vision-language tasks.}
}