Downloads · 30 days
0
llama-33b-lxctx-PI-16384-LoRA
bhenrym14/llama-33b-lxctx-PI-16384-LoRA
llama-33b-lxctx-PI-16384-LoRA is a machine learning model from bhenrym14. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Access
Public
Updated Jul 17, 2023
Repo size
975 MB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin975 MB · 100%
At a glance
- Access
- Public
- Created
- Jul 17, 2023
- Updated
- Jul 17, 2023
- SHA
- 1a76c5b6
Model card
From the Hugging Face model README
Mostly untested!
RoPE Scaled QLoRA Long Context Extension of Llama-33b (LoRA)
Overview
This is base Llama-33b with minimal additional training to extend the useful context window.
- Context length extended to 16384 by RoPE Scaled Embeddings (Position Interpolation).
- Pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
- The merged model is used as the starting point for training bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA
This is a QLoRA fine-tune
Pretraining took 10 hours on 1x RTX 6000 Ada.
- Created
- Jul 17, 2023
- Updated
- Jul 17, 2023