mpt-7b-instruct2-QLoRa-medical-QA

Model Card for mpt-7b-instruct2-QLoRa-medical-QA

<font color="FF0000" size="5"> <b> This is a QA model for answering medical questions<br /> </b></font> <br><b>Foundation Model : https://huggingface.co/ibm/mpt-7b-instruct2 <br /> Dataset : https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max <br /></b> The model has been fine tuned with 2 x GPU T4 (RAM : 2 x 14.8GB) + CPU (RAM : 29GB). <br />

<b>Model Details</b>

The model is based upon the foundation model : ibm/mpt-7b-instruct2 (Apache 2.0 License).<br /> It has been tuned with Supervised Fine-tuning Trainer and PEFT LoRa.<br />

Librairies

<ul> <li>bitsandbytes</li> <li>einops</li> <li>peft</li> <li>trl</li> <li>datasets</li> <li>transformers</li> <li>torch</li> </ul>

Notebook used for the training

You can find it in the files and versions tab or : https://colab.research.google.com/drive/14nxSP5UuJcnIJtEERyk5nehBL3W03FR3?hl=fr

=> Improvements can be achieved by increasing the number of steps and using the full dataset. <br />

Direct Use

<b>Bias, Risks, and Limitations</b>

In order to reduce training duration, the model has been trained only with the first 5100 rows of the dataset.<br />

<font color="FF0000"> Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.<br /> Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.<br /> </font>

<b>Training Details</b>

<ul> <li>per_device_train_batch_size = 1</li> <li>gradient_accumulation_steps = 16</li> <li>epoch = 5</li> <li>2 x GPU T4 (RAM : 14.8GB) + CPU (RAM : 29GB)</li> </ul>

Training Data

https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max

Training Hyperparameters

Times

Training duration : 6287.4s