Downloads · 30 days
3
5% of all-time downloads
sosoai/hansoldeco-beomi-llama3-open-ko-8b-64k-test
hansoldeco-beomi-llama3-open-ko-8b-64k-test is a text generation model from sosoai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
3
5% of all-time downloads
All-time downloads
65
Public
Parameters
8B
16.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
axolotl version: 0.4.0
base_model: beomi/Llama-3-Open-Ko-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: sosoai/mixed_dataset
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./beomi-llama3-8b-64k
save_safetensors: true
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: false
use_pose: true
pose_max_context_len: 65536
overrides_of_model_config:
rope_theta: 500000.0
max_position_embeddings: 65536
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 8
micro_batch_size: 1
num_epochs: 3
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5
train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
eval_sample_packing: False
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
pad_token: <|end_of_text|>
</details><br>
This model is a fine-tuned version of beomi/Llama-3-Open-Ko-8B on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.6869 | 0.06 | 1 | 1.7410 |
| 1.6246 | 0.52 | 9 | 1.6575 |
| 1.4583 | 1.01 | 18 | 1.4841 |
| 1.3375 | 1.53 | 27 | 1.3299 |
| 1.171 | 2.01 | 36 | 1.1744 |
| 1.0133 | 2.53 | 45 | 0.9960 |