MedAlpaca-SymptomDiagnosis

MedAlpaca-7B Fine-tuned for Medical Diagnosis

Model Description

This is a fine-tuned version of medalpaca/medalpaca-7b specialized for medical diagnosis classification based on patient symptoms.

The model was fine-tuned using LoRA (Low-Rank Adaptation) on a dataset of 10,000 medical cases with symptom-diagnosis pairs, achieving 99.0% training accuracy at checkpoint-600.

Key Features

• 🎯 High Accuracy: 99.0% training accuracy
• 🔬 Medical Specialization: Built on MedAlpaca, already optimized for medical domain
• ⚡ Efficient: Uses LoRA adapters (~67M parameters) on MedAlpaca-7B base
• 💾 Memory Efficient: Compatible with 4-bit quantization

Training Details

Base Model

• Model: medalpaca/medalpaca-7b
• Architecture: LLaMA-based medical language model
• Pre-training: Medical domain knowledge

Dataset

• Size: 10,000 symptom-diagnosis pairs
• Format: Patient symptoms → Medical diagnosis
• Train/Validation Split: Standard split with held-out validation set

Training Configuration

LoRA Hyperparameters:

• LoRA Rank (r): 16
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Target Modules: q_proj, v_proj

Training Hyperparameters:

• Learning Rate: 2e-4
• Batch Size: 4 per device
• Gradient Accumulation Steps: 4
• Effective Batch Size: 16
• Number of Epochs: 3
• Warmup Steps: 100
• Optimizer: AdamW (8-bit)
• Weight Decay: 0.01
• Max Gradient Norm: 1.0
• LR Scheduler: Linear with warmup

Training Environment:

• GPU: NVIDIA A100 40GB
• Precision: Mixed FP16
• Quantization: 4-bit NF4 with double quantization
• Framework: Hugging Face Transformers 4.45.0 + PEFT 0.12.0

Training Results

Final Checkpoint (checkpoint-600):

• Training Loss: ~0.05
• Training Accuracy: 99.0%
• Total Training Steps: 600
• Training Time: ~2 hours

Model Architecture

• Base Model: medalpaca/medalpaca-7b (~7B parameters)
• LoRA Adapters: ~67M trainable parameters
• Model Size: ~64 MB (LoRA adapters only)
• Architecture Type: LLaMA-based medical language model

Intended Use

Primary Use Cases

✅ Medical diagnosis prediction from symptom lists
✅ Clinical decision support systems (with medical oversight)
✅ Medical education and training
✅ Healthcare AI research
✅ Ensemble medical diagnosis systems

Out of Scope

❌ Direct patient care without medical professional oversight
❌ Emergency medical decisions
❌ Replacement for professional medical judgment

Usage

Installation

pip install transformers peft torch bitsandbytes accelerate

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

# Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "medalpaca/medalpaca-7b",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Load fine-tuned adapter
model = PeftModel.from_pretrained(
    base_model,
    "Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("medalpaca/medalpaca-7b")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Inference Example

# Suppress warnings
import warnings
warnings.filterwarnings('ignore')
import os
os.environ['TRANSFORMERS_VERBOSITY'] = 'error'

# Patient symptoms
symptoms = "fever, cough, fatigue, body aches, headache"

# Format prompt
prompt = f"""Below is a patient case with symptoms. Provide ONLY the most likely diagnosis.

### Symptoms:
{symptoms}

### Diagnosis:
"""

# Generate
inputs = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        temperature=0.1,
        do_sample=False
    )

# Decode
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
diagnosis = result[len(prompt):].strip().split('\n')[0]

print(f"Diagnosis: {diagnosis}")

Performance Metrics

Model Comparison

Part of a 4-model ensemble for medical diagnosis:

Limitations

• Limited to diagnostic categories in 10K training samples
• Performance depends on accurate symptom description
• Does not consider patient history, labs, or imaging
• May not perform well on rare conditions
• English language only
• Requires medical professional interpretation

Ethical Considerations

⚠️ Medical AI Ethics:

• Should never replace professional medical judgment
• Requires appropriate medical oversight in clinical settings
• Users must understand model limitations
• Clinical validation needed for real-world deployment

⚠️ Bias Considerations:

• Training data may reflect diagnostic biases
• Performance may vary across demographics
• Regular monitoring recommended for production use

Citation

If you use this model, please cite:

@misc{chauhan2024medalpaca,
  author = {Sugandha Chauhan},
  title = {MedAlpaca-7B Fine-tuned for Medical Diagnosis},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis}}
}

Disclaimer

⚠️ IMPORTANT MEDICAL DISCLAIMER

This AI model is for educational and research purposes only. It is NOT:

• A substitute for professional medical advice, diagnosis, or treatment
• Approved for clinical use without medical oversight
• Intended for emergency medical situations
• A replacement for qualified healthcare providers

Always consult a physician or qualified healthcare provider for medical decisions.

License

Released under the same license as MedAlpaca-7B. See base model for details.

Contact

Author: Sugandha Chauhan
Repository: https://huggingface.co/Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis
Model Portfolio: 4-model ensemble for medical diagnosis

---

Model Version: 1.0 (Checkpoint-600)
Last Updated: November 2024
Part of: Multi-model Medical Diagnosis System