Downloads · 30 days
80
21% of all-time downloads
leesm/llama-3-Korean-Bllossom-8B-trexlab-oki10p
llama-3-Korean-Bllossom-8B-trexlab-oki10p is a text generation model from leesm. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
- Base Model : MLP-KTLim/llama-3-Korean-Bllossom-8B - Dataset : heegyu/open-korean-instructions (10%) - Tuning Method - PEFT(Parameter Efficient Fine-Tuning) - LoRA(Low-Rank Adaptation of Large Language Models) - Rela…
Downloads · 30 days
80
21% of all-time downloads
All-time downloads
388
Public
Parameters
8B
16.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
peft_config = LoraConfig(
lora_alpha=16,
lora_dropout=0.1,
r=64,
bias='none',
task_type='CAUSAL_LM'
)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype='float16',
)
trainer = SFTTrainer(
model=peft_model,
train_dataset=dataset,
dataset_text_field='text',
max_seq_length=min(tokenizer.model_max_length, 2048),
tokenizer=tokenizer,
packing=True,
args=training_args
)
time taken : executed in 21h 45m 55s
TrainOutput(global_step=816, training_loss=1.718194248045192,
metrics={'train_runtime': 78354.6002,
'train_samples_per_second': 0.083,
'train_steps_per_second': 0.01,
'train_loss': 1.718194248045192,
'epoch': 2.99})