Downloads · 30 days
8
10% of all-time downloads
raven-lora
adriantheuma/raven-lora
raven-lora is a machine learning model from adriantheuma. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
Prompt tokenisation: LlamaTokenizer. Maximum context length: 1,204 tokens Per device train batch: 1 Gradient accumulation: 128 steps (achieving the equivalent batchsize of 128) Quantisation: 8-bit Optimiser: adamw Lea…
All-time downloads
80
Public
Repo size
105 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin105 MB · 100%
At a glance
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Jan 22, 2024
- Updated
- Jan 25, 2024
- SHA
- 05c26a83
Datasets
Model card
From the Hugging Face model README
Training details
-
Prompt tokenisation: LlamaTokenizer.
-
Maximum context length: 1,204 tokens
-
Per device train batch: 1
-
Gradient accumulation: 128 steps (achieving the equivalent batch_size of 128)
-
Quantisation: 8-bit
-
Optimiser: adamw
-
Learning_rate: 3 × 10−4
-
warmup_steps: 100
-
epochs: 5
-
Low Rank Adaptation (LoRA)
- rank: 16
- alpha: 16
- dropout: 0.05
- target modules: q_proj, k_proj, v_proj, and o_proj
This setup reduces the trainable parameters to 26,214,400 or 0.2% of the base Llama 2 13B Chat model.
Training hardware
This model is trained on commodity hardware equipped with a:
- 13th Gen Intel(R) Core(TM) i7-13700KF CPU at 3.40 GHz
- 64 GB installed RAM
- NVIDIA GeForce RTX 4090 GPU with 24 GB onboard RAM.
The trained model consumed 100 GPU hours during training.
- Library
- peft
- License
- apache-2.0
- Languages
- en
- Created
- Jan 22, 2024
- Updated
- Jan 25, 2024