Downloads · 30 days
16
0% of all-time downloads
T3Q-ko-solar-sft-v3.0
chlee10/T3Q-ko-solar-sft-v3.0
T3Q-ko-solar-sft-v3.0 is a text generation model from chlee10. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
All-time downloads
38.7K
Public
Parameters
10.7B
42.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors42.9 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Mar 25, 2024
- Updated
- Mar 25, 2024
- SHA
- 185bb7d8
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
Update @ 2024.03.25
T3Q-ko-solar-sft-v3.0
This model is a SFT fine-tuned version of chihoonlee10/T3Q-ko-solar-dpo-v3.0
Model Developers Chihoon Lee(chlee10), T3Q
Training hyperparameters
The following hyperparameters were used during training:
# 데이터셋과 훈련 횟수와 관련된 하이퍼 파라미터
batch_size = 16
num_epochs = 1
micro_batch = 1
gradient_accumulation_steps = batch_size // micro_batch
# 훈련 방법에 대한 하이퍼 파라미터
cutoff_len = 4096
lr_scheduler = 'cosine'
warmup_ratio = 0.06 # warmup_steps = 100
learning_rate = 5e-5
optimizer = 'paged_adamw_32bit'
weight_decay = 0.01
max_grad_norm = 1.0
# LoRA config
lora_r = 16
lora_alpha = 16
lora_dropout = 0.05
lora_target_modules = ["k_proj", "v_proj","gate_proj", "down_proj", "up_proj"]
# Tokenizer에서 나오는 input값 설정 옵션
train_on_inputs = False
add_eos_token = False
# NEFTune params
neftune_noise_alpha = 5
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Featherless Ai
- live
- Created
- Mar 25, 2024
- Updated
- Mar 25, 2024