Downloads · 30 days
0
alpaca-7b-lora
yahma/alpaca-7b-lora
alpaca-7b-lora is a machine learning model from yahma. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
This repo contains a low-rank adapter for LLaMA-7b fit on the Cleaned Alpaca dataset (with the new GPT-4 training data).
Access
Public
Updated Apr 10, 2023
Repo size
151 MB
Likes
13
Public
Repo makeup
Click a slice to open those files.
.bin67.2 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Apr 2, 2023
- Updated
- Apr 10, 2023
- SHA
- 66847768
Datasets
Model card
From the Hugging Face model README
This repo contains a low-rank adapter for LLaMA-7b fit on the Cleaned Alpaca dataset (with the new GPT-4 training data).
This version of the weights was trained with the following hyperparameters:
Cleaned dataset: Snapshot April 8, 2023
Epochs: 6 (Checkpoint with lowest eval loss at 3.6 epochs uploaded here)
Validation set size: 1500
Batch size: 128
Micro batch size: 8
Cutoff length: 512
Learning rate: 3e-4
Lora r: 16
Lora target modules: q_proj, k_proj, v_proj, o_proj
That is:
python finetune.py
--base_model='yahma/llama-7b-hf'
--data_path 'yahma/alpaca-cleaned'
--num_epochs=6
--cutoff_len=512
--output_dir='./lora-alpaca'
--lora_target_modules='[q_proj,k_proj, v_proj, o_proj]'
--lora_r=16
--val_set_size 1500
--micro_batch_size=8
- License
- mit
- Created
- Apr 2, 2023
- Updated
- Apr 10, 2023