Downloads · 30 days
0
llama-13b-stepwise-embeddings
Birchlabs/llama-13b-stepwise-embeddings
llama-13b-stepwise-embeddings is a machine learning model from Birchlabs. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Fine-tuned input (embedtokens: Embedding) and output (lmhead: Linear) embeddings layers, for use with Birchlabs/llama-13b-stepwise-adapter.
Access
Public
Updated Jul 10, 2023
Repo size
655 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt655 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 10, 2023
- Updated
- Jul 10, 2023
- SHA
- 2dd361bd
Model card
From the Hugging Face model README
Fine-tuned input (embed_tokens: Embedding) and output (lm_head: Linear) embeddings layers, for use with Birchlabs/llama-13b-stepwise-adapter.
Prior to finetuning: we grew the vocabulary of the tokenizer and embeddings layers. The new embeddings were average-initialized, and needed training, so we trained them. These are the weights from that training.
Ordinarily a QLoRA finetune of an LLM would not finetune the embed_tokens: Embedding (you'd need to get a bit creative, because not only have the dimensions changed, but also I don't believe any way has been established to train adapters over Embeddings).
Nor apparently would it finetune lm_head: Linear. This is harder than it sounds (i.e. you can't handle it the same way you adapt the other Linear layers), because the dimensions have grown.
- License
- apache-2.0
- Created
- Jul 10, 2023
- Updated
- Jul 10, 2023