MedSLM-SFT-LoRA

MedSLM-SFT-LoRA -- LoRA Adapters for Medical Instruction Tuning

Research Only -- Not for Clinical Use

This model is intended for research and educational purposes only. It must not be used for medical diagnosis, treatment recommendations, or any clinical decision-making.

---

Overview

This repository contains the LoRA adapter weights (~17.8 MB) produced by supervised fine-tuning (SFT) of the Saminx22/MedSLM base model on medical question-answering data. The adapters can be loaded on top of the base model using the PEFT library.

If you prefer a ready-to-use model that does not require PEFT at inference time, see the merged version: Saminx22/MedSLM-SFT.

Model Details

LoRA Configuration

Architecture

The base model uses a LLaMA-style transformer architecture:

• RMSNorm pre-normalization
• Rotary Positional Embeddings (RoPE)
• SwiGLU activation in the feed-forward network
• Grouped-Query Attention (GQA) with 16 query heads and 8 key-value heads

The base model was pre-trained from scratch on ~148M tokens of medical text (PubMed abstracts, PMC full texts, and clinical guidelines).

Training Details

Dataset

• Repository: Saminx22/medical_data_for_slm_SFT
• Splits: 46,166 train / 2,565 validation / 2,565 test
• Sources: WikiDoc, medical Q&A corpora
• Average length: ~180 tokens per example

Prompt Template

The model was trained with the following instruction template. You must use this exact format at inference time for best results:

### System:
You are a medical AI assistant. Provide accurate, evidence-based answers to medical questions.

### User:
{question}

### Assistant:
{answer}

Hyperparameters

Training Results

How to Use

Requirements

pip install transformers torch peft accelerate bitsandbytes

Loading the LoRA Adapters with PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL_ID = "Saminx22/MedSLM"
LORA_ADAPTER_ID = "Saminx22/MedSLM-SFT-LoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base_model, LORA_ADAPTER_ID)
model.eval()

Generating a Response

SYSTEM_PROMPT = (
    "You are a medical AI assistant. "
    "Provide accurate, evidence-based answers to medical questions."
)

def ask(question: str, max_new_tokens: int = 300) -> str:
    prompt = (
        f"### System:\n{SYSTEM_PROMPT}\n\n"
        f"### User:\n{question}\n\n"
        f"### Assistant:\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            top_k=50,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )

    response = output_ids[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(response, skip_special_tokens=True).strip()

print(ask("What are the warning signs of a stroke?"))

Merging Adapters into the Base Model

If you want a standalone model without the PEFT dependency at inference time, you can merge the adapters:

merged_model = model.merge_and_unload()
merged_model.save_pretrained("MedSLM-SFT-merged")
tokenizer.save_pretrained("MedSLM-SFT-merged")

Alternatively, use the pre-merged version directly: Saminx22/MedSLM-SFT.

Repository Contents

Limitations and Risks

• Research only -- not validated for clinical use or patient care.
• Small model size (~330M parameters); more prone to hallucinations and factual errors than larger models.
• No RLHF, DPO, or other safety alignment has been applied.
• Trained for single-turn question answering only; not designed for multi-turn dialogue.
• Context length limited to 1,024 tokens.
• Training data is English-only; performance on other languages is not expected.

Citation

@misc{medslm-sft-lora-2025,
  title   = {MedSLM-SFT-LoRA: LoRA Adapters for Medical Instruction Tuning},
  author  = {Saminx22},
  year    = {2025},
  publisher = {Hugging Face},
  url     = {https://huggingface.co/Saminx22/MedSLM-SFT-LoRA}
}

Related Repositories