Downloads · 30 days
10
16% of all-time downloads
Onuii/kanana-cpt-qlora-sft-lora-digging-250409
kanana-cpt-qlora-sft-lora-digging-250409 is a text generation model from Onuii. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
- Developed by: Onuii - License: apache-2.0 - Finetuned from model : Onuii/DAMI-base-merge-0408
Downloads · 30 days
10
16% of all-time downloads
All-time downloads
63
Public
Parameters
2.1B
8.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.bin4.2 GB · 50%
From the Hugging Face model README
Developed by: Onuii
License: apache-2.0
Finetuned from model : Onuii/DAMI-base-merge-0408
--run_name "DAMI-lora-sft-digging-0409"
--model_name "Onuii/DAMI-base-merge-0408"
--dataset_name "Onuii/DAMI-lecture-DB-Dataset-0408-digging"
--max_seq_length 8192
--dtype "bfloat16"
--load_in_4bit False
--r 16
--lora_alpha 32
--lora_dropout 0.1
--use_rslora True
--batch_size 16
--gradient_accumulation_steps 2
--learning_rate 1e-5
--warmup_ratio 0.05
--num_train_epochs 2
--optimizer "adamw_8bit"
--weight_decay 0.01
--lr_scheduler_type "linear"
--output_dir "../outputs"
--logging_dir "../logs"
--logging_steps 1
--save_steps 200
--save_total_limit 2
--report_to "wandb"
--use_gradient_checkpointing False
--save_strategy "steps"
--random_seed 562
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.