Downloads · 30 days
0
llygma-13b-lora
ausboss/llygma-13b-lora
llygma-13b-lora is a machine learning model from ausboss. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Access
Public
Updated May 9, 2023
Repo size
78.8 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin52.5 MB · 100%
At a glance
- Access
- Public
- Created
- May 3, 2023
- Updated
- May 9, 2023
- SHA
- e181f244
Model card
From the Hugging Face model README
training_arguments = transformers.TrainingArguments(
per_device_train_batch_size=4,
gradient_accumulation_steps=BATCH_SIZE // MICRO_BATCH_SIZE,
warmup_steps=100,
max_steps=300,
learning_rate=3e-4,
fp16=True,
logging_steps=10,
optim="adamw_torch",
evaluation_strategy="steps",
save_strategy="steps",
eval_steps=50,
save_steps=50,
save_total_limit=3,
load_best_model_at_end=True,
)
LORA_R = 8
LORA_ALPHA = 16
LORA_DROPOUT= 0
LORA_TARGET_MODULES = [
"q_proj",
"v_proj",
]
BATCH_SIZE = 128
MICRO_BATCH_SIZE = 4
GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
LEARNING_RATE = 3e-4
TRAIN_STEPS = 300
- Created
- May 3, 2023
- Updated
- May 9, 2023