Downloads · 30 days
0
Llama-3-8B-lora
geshijoker/Llama-3-8B-lora
Llama-3-8B-lora is a machine learning model from geshijoker. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
llama3-8B supervised finetuning with lora
Access
Public
Updated Sep 5, 2024
Repo size
13.6 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors13.6 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024
- SHA
- e2f1fe82
Model card
From the Hugging Face model README
Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->llama3-8B supervised finetuning with lora
Model Details
0:"q_proj" 1:"v_proj" lora_alpha:32 lora_dropout:0.05 r:8 gamma:0.85 batch_size_training:4 gradient_accumulation_steps:4 lr:0.0001 num_epochs:3 optimizer:"AdamW" peft_method:"llama_adapter" trainable params: 3,407,872 || all params: 8,033,669,120 || trainable%: 0.0424
Model Description
Average epoch time: 781s Train loss: 0.2921483516693115 Eval loss: 1.125542402267456 Eval perplexity: 3.08188796043396
Max CUDA memory allocated was 48 GB Max CUDA memory reserved was 55 GB Peak active CUDA memory was 48 GB CUDA Malloc retries : 0 CPU Total Peak Memory consumed during the train (max): 3 GB
- Library
- transformers
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024