Downloads · 30 days
5
4% of all-time downloads
richard-park/llama-3.1-fineweb-edu-1m-Inst
llama-3.1-fineweb-edu-1m-Inst is a text generation model from richard-park. Use it when you need the model to write or continue text. It is set up for transformers.
- Use default tokenizer - 각 단계별 훈련 후, embedtokens와 lmhead도 병합 - phase <details <summary hyperparameter in phase</summary
Downloads · 30 days
5
4% of all-time downloads
All-time downloads
138
Public
Parameters
8B
16.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
pre1
hyperparameter test: rank=16 lr=1e-5 weight_decay=0.01 drop_out=0.1 dataset: richard-park/llama-recipe-pre1-fineweb-edu-1m-split-text llama-3.1 Base Model llama recipe 1 훈련
pre2
hyperparameter 변경: lr= 1e-5->2e-6, epoch: 3->4, lora dropout: 0.1 -> 0.3
dataset: richard-park/llama-recipe-pre2-fineweb-edu-1m-split-text
llama-3.1 Base Model llama recipe 2 훈련
nohup tensorboard --logdir=sapie-fineweb-edu/outputs/pre2/llama31-base-1m-fineweb-edu_20250128-01/runs --host 0.0.0.0 --port=5406 > tensorboard.log 2>&1 & disown
pre3
hyperparameter: rank=16/32, lr=1e-5, weight_decay=0.1 drop_out=0.3
dataset: richard-park/sapie-dataset-pre3-1m-gt50-le256-split
llama-3.1 Base Model llama recipe 3 훈련
nohup tensorboard --logdir=sapie-fineweb-edu/outputs/pre3/llama31-base-1m-aihub-trans_20250130-01/runs --host 0.0.0.0 --port=5406 > tensorboard.log 2>&1 & disown
This is a merge of pre-trained language models created using mergekit.
This model was merged using the DARE TIES merge method using ../models/Llama-3.1-8B-Instruct as a base.
The following models were included in the merge:
The following YAML configuration was used to produce this model:
models:
- model: ../models/Llama-3.1-8B-Instruct
parameters:
density: [0.6, 0.8, 1] # 하위→중간→상위 순, 상위 보존
weight:
- filter: mlp
value: 0.8 # MLP 레이어에 더 큰 가중치 (출력에 기여)
- value: 0.5 # 나머지 레이어
- model: ../models/llama31-base-pre3-finweb-edu-1m # base model + 한국어 pretrain
parameters:
density: [1, 0.6, 0.4] # 하위에 더 큰 영향을 주도록 설정
weight:
- filter: attention
value: 0.7 # Attention 레이어에 더 큰 가중치 (한국어 학습 보강)
- value: 0.3 # 나머지 레이어
merge_method: dare_ties
base_model: ../models/Llama-3.1-8B-Instruct
dtype: bfloat16
- {'answer_relevancy': 0.7495, 'faithfulness': 0.6831}
- {'answer_relevancy': 0.7406, 'faithfulness': 0.7337}
- {'answer_relevancy': 0.7356, 'faithfulness': 0.6814}