MedLam

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This Llama 3.1 8B Instruct Model was obtained using the Unsloth library. It is fine-tuned using the LoRA PEFT approach to give better responses to medical questions.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

MedLam explores the fine-tuning of LLMs using a Medical QA dataset to improve their performance for domain-specific tasks. By combining state-of-the-art natural language processing techniques and medical data, MedLam aims to deliver an effective and intuitive medical assistant.

• Developed by: Avishek Choudhury

<!-- - **Funded by [optional]:** [More Information Needed] --> <!-- - **Shared by [optional]:** [More Information Needed] -->

• Model type: Transformer based auto-regressive language model
• Language(s) (NLP): English
• License: MIT
• Finetuned from model Meta Llama 3.1 8B Instruct: unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/avishek04/MedLam/tree/main
• Paper: https://arxiv.org/abs/2106.09685

<!-- - **Demo [optional]:** [More Information Needed] -->

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

This model was developed only for research and learning purposes and is not meant for commercialization as a final product.

• Assisting medical students in understanding complex topics.
• Supporting professors in teaching and curriculum design.
• Helping doctors with quick and accurate responses to medical questions.

How to Get Started with the Model

Use the code below to get started with the model.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit")
model = PeftModel.from_pretrained(base_model, "aviici4cs/MedLam")

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

1. Data Preparation:
   • Preprocessed Medical QA datasets for training and evaluation.
2. Model Fine-Tuning:
   • Applied LoRA PEFT (Parameter-Efficient Fine-Tuning) to train LLMs on domain-specific data.
3. Performance Validation:
   • Analyzed and compared models across multiple configurations to determine optimal hyperparameters.
4. Deployment (Future Scope):
   • Aimed at integrating the fine-tuned model into a user-friendly medical assistant platform.

Hardware

Google Colab

Software

• Programming Languages: Python
• Frameworks: Unsloth Library, PyTorch, Hugging Face Transformers
• Machine Learning Techniques: LoRA PEFT, Deep Learning
• Tools: NumPy, Pandas, Sci-kit Learn

Model Card Contact

• https://www.linkedin.com/in/avishekchoudhury/
• https://github.com/avishek04
• https://avishek04.github.io/

Framework versions

• PEFT 0.13.2