Downloads · 30 days
0
RWKV-LM-LoRA-Alpaca-Cleaned-Japan
shi3z/RWKV-LM-LoRA-Alpaca-Cleaned-Japan
RWKV-LM-LoRA-Alpaca-Cleaned-Japan is a machine learning model from shi3z. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-4.0.
BaseModel: RWKV-4-Pile-14B-Instruct-test5-20230329-ctx4096.pth URL: https://huggingface.co/BlinkDL/rwkv-4-pile-14b
Access
Public
Updated May 7, 2023
Repo size
146 MB
Likes
9
Public
Repo makeup
Click a slice to open those files.
.pth146 MB · 100%
At a glance
- License
- cc-by-nc-4.0
- Access
- Public
- Created
- Apr 26, 2023
- Updated
- May 7, 2023
- SHA
- 87cffa18
Model card
From the Hugging Face model README
BaseModel: RWKV-4-Pile-14B-Instruct-test5-20230329-ctx4096.pth URL: https://huggingface.co/BlinkDL/rwkv-4-pile-14b
Training Code: https://github.com/Blealtan/RWKV-LM-LoRA
Dataset: https://github.com/shi3z/alpaca_ja
Loss: Epoch 320: loss 0.76 Epoch 500: loss 0.62
Training Prompt:
python train.py
--load_model RWKV-4-Pile-14B-Instruct-test5-20230329-ctx4096.pth
--proj_dir out
--data_file "train.npy"
--data_type "numpy"
--vocab_size 50277
--ctx_len 1024
--epoch_save 5
--epoch_count 100
--n_layer 40
--n_embd 5120
--epoch_steps 1000 --epoch_begin 0 --micro_bsz 1 --pre_ffn 0 --head_qk 0 --lr_init 1e-5 --lr_final 1e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.999 --adam_eps 1e-8 --accelerator gpu --devices 1 --precision bf16 --strategy deepspeed_stage_2 --grad_cp 0
--lora --lora_r 8 --lora_alpha 32 --lora_dropout 0.01
Training Environment: A6000x1 + XEON 256GB RAM
- License
- cc-by-nc-4.0
- Languages
- ja
- Created
- Apr 26, 2023
- Updated
- May 7, 2023