GPT-2-LoRA-HealthCare

GPT-2-LoRA-HealthCare

Original Author: BastienHot ZhanPascal<br> Date created: 2024/02/25 <br> Dataset:

<ul> <li><a href="https://www.kaggle.com/datasets/jpmiller/layoutlm">MedQuAD from Kaggle</a></li> <li><a href="https://pubmedqa.github.io/">PubMedQA</a></li> <li><a href="http://participants-area.bioasq.org/datasets/">BioASQ – Task B</a></li> <li><a href="https://quac.ai/">QuAC</a></li> <li><a href="https://rajpurkar.github.io/SQuAD-explorer/">SQuAD</a></li> </ul>

Spaces Demo

Description : GPT-2-LoRA-HealthCare

The GPT-2-LoRA-HealthCare model was developed as part of a student project during the Bachelor of Technology (BUT) in Computer Science at IUT Villetaneuse. It is based on the pre-trained model from keras_nlp (gpt2-large-en), with the incorporation of the LoRA technique to enhance training efficiency. The model is specifically designed for Q&A interactions in a healthcare context, with a patient asking a question and the model responding with an appropriate answer.

LoRA is a training technique used for large language models (LLM) to train them more efficiently and with a less time consuming approach. How it works is the following:

<ul> <li>Look at the pretrained model's weights</li> <li>Determine the linearly independent and dependent columns of the matrix</li> <li>Create the new LoRA layer with only the linearly independent matrix columns</li> <li>Freeze the pretrained model's weights</li> <li>Train the LoRA layer weights</li> <li>Merge the weights of the pretrained model and the LoRA layer</li> </ul>