Downloads · 30 days
0
airoboros-l2-13b-2.1-YaRN-64k-LoRA
bhenrym14/airoboros-l2-13b-2.1-YaRN-64k-LoRA
airoboros-l2-13b-2.1-YaRN-64k-LoRA is a machine learning model from bhenrym14. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a finetune of NousResearch/Yarn-Llama-2-13b-64k. This starting point is Llama-2-13b with additional pretraining done with YaRN scaling applied to RoPE to extend the useful context length to 64k tokens. Startin…
Access
Public
Updated Sep 3, 2023
Repo size
501 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors501 MB · 100%
At a glance
- Access
- Public
- Created
- Sep 3, 2023
- Updated
- Sep 3, 2023
- SHA
- 2a54a02e
Datasets
Model card
From the Hugging Face model README
Extended Context (via YaRN) Finetune of Llama-2-13b with airoboros-2.1 (LoRA)
Overview
This is a finetune of NousResearch/Yarn-Llama-2-13b-64k. This starting point is Llama-2-13b with additional pretraining done with YaRN scaling applied to RoPE to extend the useful context length to 64k tokens. Starting with this model, I performed instruction tuning with Jon Durbin's Airoboros 2.1 dataset, with same scaling approach applied.
This is a (merged) QLoRA fine-tune (rank 64).
The finetune was performed with 1x RTX 6000 Ada (~18 hours).
For full model card, including benchmarks, see the model card of the fp16 merged model
- Created
- Sep 3, 2023
- Updated
- Sep 3, 2023