fine-tuned-bert

Model Description

<p>This model is a fine tuned version of the <a href="https://huggingface.co/distilbert/distilbert-base-uncased">distilbert-base-uncased</a> model on Hugging face. The model is trained to classify payment notes for business owners into one of the following categories.</p> <ol> <li>INVENTORY, SUPPLIES AND EQUIPMENT</li> <li>PROFESSIONAL SERVICES</li> <li>TRANSPORTATION AND TRAVEL</li> <li>UTILITIES</li> <li>EMPLOYEE BENEFITS AND COMPENSATION</li> <li>MEALS AND ENTERTAINMENT</li> <li>TAX PAYMENTS</li> <li>LEGAL AND COMPLIANCE FEES</li> <li>BUSINESS DEVELOPMENT AND INVESTMENT</li> </ol>

Base Model Description

<p>DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a self-supervised fashion, using the BERT base model as a teacher. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts using the BERT base model.</p>

Training results

<table> <tr> <th>Epoch</th> <th>Training Loss</th> <th>Validation Loss</th> <th>Accuracy</th> </tr> <tr> <th>0</th> <th>No Log</th> <th>0.263793</th> <th>0.916230</th> </tr> <tr> <th>1</th> <th>No Log</th> <th>0.185122</th> <th>0.937173</th> </tr> <tr> <th>2</th> <th>0.318300</th> <th>0.191695</th> <th>0.937173</th> </tr> </table>

Training results

<p>Check out the training code at this <a href="https://github.com/samanthaKarungi/iotec-pay-model-bert/tree/main/model/training_and_evaluation">github repo</a></p>

Framework versions

<ul> <li>Transformers 4.37.2</li> <li>PyTorch 2.2.0</li> <li>Datasets 2.17.1</li> <li>Tokenizers 0.15.2</li> </ul>