vitalik.eth-13b

Prerequisites

The vitalik-13b.csv file contains the QA set required for the fine-tuning. Each QA pair length is limited to 80 words to fit into llama.cpp context length of 128 tokens. The convert.py script converts the CSV file into QAs in the llama2 chat template.

python convert.py

It generates a vitalik-13b.txt file, which can now be used in fine-tuning.

Fine-tuning steps

Clone this repo into the llama.cpp/models/ folder.

cd llama.cpp/models
git clone https://huggingface.co/gaianet/vitalik.eth-13b

Move the Llama2-13b-chat base model to the folder.

cd vitalik-13b
mv path/to/llama-2-13b-chat.Q5_K_M.gguf .

From the llama.cpp/models/vitalik-13b folder run the following command.

../../build/bin/finetune --model-base llama-2-13b-chat.Q5_K_M.gguf --lora-out lora.bin --train-data vitalik-13b.txt --sample-start '<SFT>' --adam-iter 1024

Wait for several days until the above process finishes. You will have a lora.bin file, which can generate the fine-tuned model.

../../build/bin/export-lora --model-base llama-2-13b-chat.Q5_K_M.gguf --lora lora.bin --model-out vitalik.eth-13b-q5_k_m.gguf

Learn more about Llama2 model fine tuning here.