LLaMA-3.1-8B-Infinity3M-Kobo

LLaMA-3.1-8B-Infinity3M-Kobo

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B on the https://huggingface.co/datasets/KoboldAI/infinity3m-kobo dataset. With this model we hope to provide a suitable base for further fiction tunes, this tune makes use of the highly mergable alpaca format and was stripped of all writing tasks. Due to the purposeful removal of fiction related tasks this model will be unusable in the usual use cases our community enjoys, but prevents undesirable biases in fiction tunes trained on top of this instruct model.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 25
• num_epochs: 3.0

Training results

Framework versions

• Transformers 4.43.4
• Pytorch 2.4.0
• Datasets 2.20.0
• Tokenizers 0.19.1

Special thanks to G4rg for the compute!