Downloads · 30 days
0
Llama-3-8B-llama-adapter-adapter-4bit
geshijoker/Llama-3-8B-llama-adapter-adapter-4bit
Llama-3-8B-llama-adapter-adapter-4bit is a machine learning model from geshijoker. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
llama3-8B supervised finetuning with llama-adapter 4bit quantization
Access
Public
Updated Sep 5, 2024
Repo size
4.9 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.9 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024
- SHA
- 962ab564
Model card
From the Hugging Face model README
Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->llama3-8B supervised finetuning with llama-adapter 4bit quantization
Model Details
adapter_layers:30 adapter_len:10 gamma:0.85 batch_size_training:4 gradient_accumulation_steps:4 lr:0.0001 num_epochs:3 num_freeze_layers:1 optimizer:"AdamW" peft_method:"llama_adapter" trainable params: 1,228,830 || all params: 8,031,490,078 || trainable%: 0.0153
Model Description
Average epoch time: 566s Train loss: 0.41620415449142456 Eval loss: 1.57061767578125
Max CUDA memory allocated was 14 GB Max CUDA memory reserved was 16 GB Peak active CUDA memory was 14 GB CPU Total Peak Memory consumed during the train (max): 4 GB
- Library
- transformers
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024