BasicEconomics-SpicyBoros-2.2-7B-QLORA-v0.2-GGUF

GGUF files for the below.

QLORA on SpicyBoros 2.2 Llama 2 7B using synthetic Q&A Dataset. 3 epochs on the 2nd version of the dataset that has been augumented with more details about Thomas' personal life. There is still room for improvement with prompt formatting

My intention with this QLORA is mostly to try to train something usable and cool locally on normal desktop without going to runpod. I tried training q4_0 quant with cpu-lora in llama.cpp (https://rentry.org/cpu-lora) but it's been a miss, it's about 20x slower on 11400f than on poorman's GTX 1080.

The model can be used to ask questions about basic economic concepts, responses will have a viewpoint similar to the one expressed by Thomas Sowell in his book Basic Economics. Version 0.2 is much better than 0.1. The dataset was formatted better, it's bigger and it was trained for 3 epochs as oposed to 0.95 epoch previously.

Prompt format that works well:

A chat between Reader and Thomas Sowell.

Reader: {prompt} Thomas: {response}

I am doing interference with koboldcpp so I am putting "A chat between Reader and Thomas Sowell." in Memory tab. There are no whitespaces before Reader:, one whitespace after Reader:, and one whitespace before and one after for Thomas: I was training on the sequence length of 1024, but I conversed with the model up to 4000 tokens and it was still coherent and in character. Even though the training date I used is only single turn, model has no issue with multi-turn conversations. Much of that is thanks to the fine-tuning done earlier by amazing Jon Durbin.

Known issues:

Prompt formatting could be a bit better, sometimes the response is out of character if you don't put the "A chat between Reader and Thomas Sowell." to memory. Also, I would like for responses to be more like Thomas Sowell while on interviews than how he writes books.