Downloads · 30 days
34
27% of all-time downloads
Llama-3.1-Discord-Short-8B
mpasila/Llama-3.1-Discord-Short-8B
Llama-3.1-Discord-Short-8B is a text generation model from mpasila. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
Trained on Discord chatlogs from this dataset.
All-time downloads
128
Public
Parameters
8B
16.1 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors16.1 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- llama3.1
- Model type
- llama
- Access
- Public
- Created
- Sep 14, 2024
- Updated
- Sep 15, 2024
- SHA
- f2f6a356
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
Trained on Discord chatlogs from this dataset.
Uses Llama 3.1 formatting.
LoRA: mpasila/Llama-3.1-Discord-Short-LoRA-8B
Trained with regular LoRA (not quantized/QLoRA) and LoRA rank was 128 and Alpha set to 32. Trained for 1 epoch using A40 for about 5,5 hours.
args = UnslothTrainingArguments(
per_device_train_batch_size = 1,
gradient_accumulation_steps = 8,
warmup_ratio = 0.1,
num_train_epochs = 1,
learning_rate = 5e-5,
embedding_learning_rate = 5e-6,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.00,
lr_scheduler_type = "cosine",
seed = 3407,
output_dir = "outputs",
),
Uploaded model
- Developed by: mpasila
- License: Llama 3.1 Community License Agreement
- Finetuned from model : unsloth/Meta-Llama-3.1-8B
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama3.1
- Languages
- en
- Created
- Sep 14, 2024
- Updated
- Sep 15, 2024