Downloads · 30 days
421
3% of all-time downloads
RichardErkhov/Na0s_-_Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0-gguf
Na0s_-_Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
421
3% of all-time downloads
All-time downloads
13K
Public
Repo size
80.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf93.3 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0 - GGUF
library_name: transformers datasets:
<a href="https://ibb.co/xGnXR5t"><img src="https://i.ibb.co/P42FfCv/DALL-E-2024-08-08-05-21-39-An-artistic-representation-for-a-model-card-featuring-an-abstract-and-sty.webp" alt="DALL-E-2024-08-08-05-21-39-An-artistic-representation-for-a-model-card-featuring-an-abstract-and-sty" border="0"></a>
model = FastLanguageModel.get_peft_model(
model,
r = 4,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 4,
lora_dropout = 0.05,
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 3407,
use_rslora = False,
loftq_config = None,
)
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "completion",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
packing = False,
args = TrainingArguments(
per_device_train_batch_size = 10,
gradient_accumulation_steps = 4,
warmup_steps = 5,
max_steps=5000,
learning_rate = 2e-4,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "cosine",
seed = 3407,
output_dir = "outputs_4",
push_to_hub=True,
hub_always_push=True,
),
)
Load Mode Memory Metrics
| Model | Max Global VRAM (MB) | Max Process VRAM (MB) | Max Reserved VRAM (MB) | Max Allocated VRAM (MB) |
|---|---|---|---|---|
| Llama-3.1-8B | 18521.98 | 16630.42 | 16196.30 | 16060.54 |
| Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0 | 16319.97 | 14428.41 | 13994.30 | 13879.42 |
Inference Mode Latency Metrics
| Model | Latency Mean (s) | Throughput (tokens/s) |
|---|---|---|
| Llama-3.1-8B | 0.8104 | 38.2536 |
| Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0 | 0.5530 | 56.0570 |
<a href="https://ibb.co/6ngkh8w"><img src="https://i.ibb.co/dWGR9tM/Screenshot-2024-08-08-at-7-41-26-AM.png" alt="Screenshot-2024-08-08-at-7-41-26-AM" border="0" width=600 height=600></a>
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).