Downloads · 30 days
0
llm-jp-3-13b-finetune
kizm0203/llm-jp-3-13b-finetune
llm-jp-3-13b-finetune is a machine learning model from kizm0203. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Access
Public
Updated Dec 16, 2024
Repo size
250 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors250 MB · 98%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Dec 16, 2024
- Updated
- Dec 16, 2024
- SHA
- d2d0a7e3
Model card
From the Hugging Face model README
llm-jp-3-13b-finetune
概要 ベースモデル: llm-jp-3-13b Ichikara Instructionデータセットを用いてファインチューニングを行いました． LoRAによってファインチューニングを行いました．
LoRA設定
LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=modules,
)
TrainingArguments(
output_dir=new_model_id,
per_device_train_batch_size=1,
gradient_accumulation_steps=2,
optim="paged_adamw_32bit",
num_train_epochs=1,
logging_strategy="steps",
logging_steps=10,
warmup_steps=10,
save_steps=100,
save_total_limit = 2,
max_steps = -1,
learning_rate=5e-5,
fp16=False,
bf16=False,
seed = 3407,
group_by_length=True,
report_to="none"
)
- Library
- transformers
- Languages
- ja
- Created
- Dec 16, 2024
- Updated
- Dec 16, 2024