Downloads · 30 days
5
4% of all-time downloads
frankmorales2020/NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata
NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata is a machine learning model from frankmorales2020. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as llama3.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
5
4% of all-time downloads
All-time downloads
129
Public
Repo size
33.6 GB
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB · 99%
From the Hugging Face model README
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B on the generator dataset. It achieves the following results on the evaluation set:
Accuracy (Eval dataset and predict) for a sample of 10: 70.00%
Fine-Tuning: https://github.com/frank-morales2020/MLxDL/blob/main/FineTuning_LLM_Meta_Llama_3_8B_for_MEDAL_EVALDATA.ipynb
Evaluation: https://github.com/frank-morales2020/MLxDL/blob/main/Meta_Llama_3_8B_for_MEDAL_EVALUATOR_evaldata.ipynb
from transformers import EarlyStoppingCallback trainer.add_callback(EarlyStoppingCallback(early_stopping_patience=5))
trainer.train()
trainer.save_model()
The following hyperparameters were used during training:
from transformers import TrainingArguments
args = TrainingArguments(
output_dir="/content/gdrive/MyDrive/model/NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata",
#num_train_epochs=3, # number of training epochs
num_train_epochs=1, # number of training epochs for POC
per_device_train_batch_size=2, # batch size per device during training
gradient_accumulation_steps=8, # number of steps before performing a backward/update pass
gradient_checkpointing=True, # use gradient checkpointing to save memory
optim="adamw_torch_fused", # use fused adamw optimizer
logging_steps=200, # log every 200 steps
learning_rate=2e-4, # learning rate, based on QLoRA paper # i used in the first model
bf16=True, # use bfloat16 precision
tf32=True, # use tf32 precision
max_grad_norm=1.0, # max gradient norm based on QLoRA paper
warmup_ratio=0.05, # warmup ratio based on QLoRA paper = 0.03
weight_decay=0.01,
lr_scheduler_type="cosine", # lr_scheduler_type="cosine" (Cosine Annealing Learning Rate)
push_to_hub=True, # push model to hub
report_to="tensorboard", # report metrics to tensorboard
gradient_checkpointing_kwargs={"use_reentrant": True},
load_best_model_at_end=True,
logging_dir="/content/gdrive/MyDrive/model/NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata/logs",
evaluation_strategy="steps", # Evaluate at step intervals
eval_steps=200, # Evaluate every 50 steps
save_strategy="steps", # Save checkpoints at step intervals
save_steps=200, # Save every 50 steps (aligned with eval_steps)
metric_for_best_model = "loss",
]
)