Downloads · 30 days
0
marianbasti/Llama-2-13b-alpaca-spanish-LoRA
Llama-2-13b-alpaca-spanish-LoRA is a text generation model from marianbasti. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
This is a LoRA for Llama 2 13B trained on a translated alpaca dataset on an attempt to improve spanish performance of the Llama-2 foundation model with a conversational focus.
Downloads · 30 days
0
Access
Public
Updated Jul 26, 2023
Repo size
1.7 GB
Likes
1
Public
Click a slice to open those files.
.bin1.7 GB · 100%
From the Hugging Face model README
This is a LoRA for Llama 2 13B trained on a translated alpaca dataset on an attempt to improve spanish performance of the Llama-2 foundation model with a conversational focus.
Base model used was The Bloke's Llama-2-13B-fp16 trained in 4bit precision with an added padding token.
The original Llama 2 model does not have a padding token, this came to be restrictive when training. To address this, I added a padding token to the tokenizer associated with the model.
from transformers import LlamaTokenizer, LlamaForCausalLM
model_name = 'TheBloke/Llama-2-13B-fp16'
model = LlamaForCausalLM.from_pretrained(model_name).half()
tokenizer = LlamaTokenizer.from_pretrained(model_name)
# Add padding token
tokenizer.add_tokens(['<PAD>'])
tokenizer.pad_token = '<PAD>'
# Resizing the model
model.resize_token_embeddings(len(tokenizer))
padded_model_name = 'Llama-2-13B-fp16-padded'
# Save
tokenizer.save_pretrained(padded_model_name)
model.save_pretrained(padded_model_name)
| Training parameteres | |
|---|---|
| LoRA scale | 2 |
| Epochs | 0.75 |
| Learning Rate | 2e-5 |
| Warmup Steps | 100 |
| Loss | 1.07 |