Yi-34B-Spicyboros-2-2-run3-QLoRA

<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

qlora-yi-34b-spicyboros-3

This fine-tune took about 10 hours with single RTX 3090 Ti 24GB on Spicyboros 2.2 dataset, 0.2 epochs.

Model description

Uncensored Spicyboros based on Yi-34B, fine-tuned on local consumer GPU. I used Llama-fied Yi-34B as base, it has layer names changed to those commonly found in models with Llama architecture.

How to Use

Merge base model Yi-34B-Llama link (llama-tokenizer branch) with the adapter.bin file from this repo. Script for merging is in config folder. Measurement.json file for Exllama v2 is in this repo too. I used spicyboros 2.2 dataset converted to parquet as a calibration dataset for the measurement.json

Intended uses & limitations

Use is limited by Yi license

Known Issues

Thesaurus mode seems to be solved by lowering repp to 1.0. I saw it going into loops of repeating sentences once or twice, but it doesn't happen all the time.

Prompt Format

ChatML prompt format is burned in. Here's a prompt format you should use

<|im_start|>system
A chat.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Training and evaluation data

I used now non-public Jon Durbin's Spicyboros 2.2 dataset. Trained on just 0.2 epochs since I already lost 4 days doing unsuccessful runs with longer training time and I got tired of that. Training config for axolotl is in this repo, it uses a preset file that is modified a little bit to fit the json formatting of the dataset I had on hand. I can share the modified preset file if you want.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 100
• num_epochs: 0.2

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1+cu118
• Datasets 2.14.5
• Tokenizers 0.14.1