Downloads · 30 days
0
karishmagrana/llama-2-finetune-7b-lora
llama-2-finetune-7b-lora is a machine learning model from karishmagrana. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
In order to execute this code, if the notebook is run in colab, it can be mount to drive, otherwise files can be accessed and saved locally.
Downloads · 30 days
0
Access
Public
Updated Dec 11, 2023
Repo size
16.8 MB
Likes
0
Public
Click a slice to open those files.
.bin16.8 MB · 98%
From the Hugging Face model README
In order to execute this code, if the notebook is run in colab, it can be mount to drive, otherwise files can be accessed and saved locally.
Once the 3rd cell for hugging face login is run, hugging face will require a login. This will require meta and hugging face access to llama-2.
The Lora weights are outlined below and are included in the notebook and just need to be run through the cell.
The Training weights are outlined below and are included in the notebook and just need to be run through the cell.
Ensure the datapath is altered as needed. During training, a checkpoint should be saved every 1000 steps.
After training a finetuned model will be output and a tensorboard showing training/epoch loss data. For every 10 steps, the hyperparameters and loss are shown in the trainer_state.json.
peft_parameters = LoraConfig( lora_alpha=16, lora_dropout=0.1, r=8, bias="none", task_type="CAUSAL_LM" )
train_params = TrainingArguments( output_dir="", num_train_epochs=3, per_device_train_batch_size=4, gradient_accumulation_steps=1, optim="paged_adamw_32bit", save_steps=1000, logging_steps=10, learning_rate=2e-5, weight_decay=0.001, fp16=False, bf16=False, max_grad_norm=0.3, max_steps=-1, warmup_ratio=0.03, group_by_length=True, lr_scheduler_type="constant", report_to="tensorboard" )
The following bitsandbytes quantization config was used during training: