Downloads · 30 days
18
9% of all-time downloads
Llama-3.1-Discord-Short-LoRA-8B
mpasila/Llama-3.1-Discord-Short-LoRA-8B
Llama-3.1-Discord-Short-LoRA-8B is a machine learning model from mpasila. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as llama3.1.
Trained on Discord chatlogs from this dataset.
All-time downloads
207
Public
Repo size
5.5 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors5.5 GB · 100%
At a glance
- Library
- peft
- License
- llama3.1
- Access
- Public
- Created
- Sep 14, 2024
- Updated
- Sep 15, 2024
- SHA
- c0e26570
Base models
Datasets
Model card
From the Hugging Face model README
Trained on Discord chatlogs from this dataset.
Uses Llama 3.1 formatting.
Merged model: mpasila/Llama-3.1-Discord-Short-8B
Trained with regular LoRA (not quantized/QLoRA) and LoRA rank was 128 and Alpha set to 32. Trained for 1 epoch using A40 for about 5,5 hours.
args = UnslothTrainingArguments(
per_device_train_batch_size = 1,
gradient_accumulation_steps = 8,
warmup_ratio = 0.1,
num_train_epochs = 1,
learning_rate = 5e-5,
embedding_learning_rate = 5e-6,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.00,
lr_scheduler_type = "cosine",
seed = 3407,
output_dir = "outputs",
),
Uploaded model
- Developed by: mpasila
- License: Llama 3.1 Community License Agreement
- Finetuned from model : unsloth/Meta-Llama-3.1-8B
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.
- Library
- peft
- License
- llama3.1
- Languages
- en
- Created
- Sep 14, 2024
- Updated
- Sep 15, 2024