Downloads · 30 days
15
4% of all-time downloads
llama2-22b-blocktriangular-alpaca
nkpz/llama2-22b-blocktriangular-alpaca
llama2-22b-blocktriangular-alpaca is a text generation model from nkpz. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
There is no official 22b model, this is just a weird experiment, and any potential benefits of doing this have not been validated
All-time downloads
345
Public
Parameters
21.8B
43.7 GB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.safetensors43.7 GB · 100%
Parameter types
How the weights are stored.
F1621.8B · 100%
Try a prompt
Model card
From the Hugging Face model README
There is no official 22b model, this is just a weird experiment, and any potential benefits of doing this have not been validated
https://huggingface.co/chargoddard/llama2-22b-blocktriangular trained one one epoch of 52k rows of Stanford Alpaca. About 11 hours on a 3090.
I had trouble with training using the other 22b method with BLOCK_DIAGONAL=True as done in https://huggingface.co/chargoddard/llama2-22b, but with this method, this is the first time I've been able to target all modules without breaking the output.
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "gate_proj", "down_proj"]
Trained at 5e-5 with r=32. For more info see https://wandb.ai/nkpz/huggingface/runs/3oy5nbtv/workspace?workspace=user-nkpz
It's been responding coherently enough that I would need to run some objective benchmarks to determine if this is better/worse than stock llama 13b
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- other
- Created
- Aug 3, 2023
- Updated
- Aug 3, 2023