Downloads · 30 days
0
Llama-3-8B-llama-adapter
geshijoker/Llama-3-8B-llama-adapter
Llama-3-8B-llama-adapter is a machine learning model from geshijoker. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
llama3-8B supervised finetuning with llama-adapter
Access
Public
Updated Sep 5, 2024
Repo size
4.9 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.9 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024
- SHA
- fc7ef47b
Model card
From the Hugging Face model README
Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->llama3-8B supervised finetuning with llama-adapter
Model Details
adapter_layers:30 adapter_len:10 gamma:0.85 batch_size_training:4 gradient_accumulation_steps:4 lr:0.0001 num_epochs:3 num_freeze_layers:1 optimizer:"AdamW" peft_method:"llama_adapter" trainable params: 1,228,830 || all params: 8,031,490,078 || trainable%: 0.0153
Model Description
Average epoch time: 967s Train loss: 0.3901134133338928 Eval loss: 1.466189980506897 Eval perplexity: 4.332696437835693
Max CUDA memory allocated was 49 GB Max CUDA memory reserved was 55 GB Peak active CUDA memory was 49 GB CPU Total Peak Memory consumed during the train (max): 4 GB
- Library
- transformers
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024