Downloads · 30 days
0
airoboros-13b-gpt4-1.4.1-PI-8192-LoRA
bhenrym14/airoboros-13b-gpt4-1.4.1-PI-8192-LoRA
airoboros-13b-gpt4-1.4.1-PI-8192-LoRA is a machine learning model from bhenrym14. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Full model card with merged GPTQ 4bit quantized weights can be found here: https://huggingface.co/bhenrym14/airoboros-13b-gpt4-1.4.1-PI-8192-GPTQ
Access
Public
Updated Jul 7, 2023
Repo size
501 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin501 MB · 100%
At a glance
- Access
- Public
- Created
- Jul 6, 2023
- Updated
- Jul 7, 2023
- SHA
- 6ce67a32
Datasets
Model card
From the Hugging Face model README
RoPE Scaled QLoRA Fine-tune of Llama-13b on airoboros-gpt4-1.4.1 (LoRA)
Full model card with merged GPTQ 4bit quantized weights can be found here: https://huggingface.co/bhenrym14/airoboros-13b-gpt4-1.4.1-PI-8192-GPTQ
fp16 merged weights can be found here: https://huggingface.co/bhenrym14/airoboros-13b-gpt4-1.4.1-PI-8192-fp16
Overview
This is Jon Durbin's Airoboros 13B GPT4 1.4 (LoRA weights) with several key modifications:
- Context length extended to 8192 by RoPE Scaled Embeddings, but NOT via the superHOT LoRA. I started with base Llama-13b.
- Training sequences beyond 2048 have the target truncated to equal 2048.
- Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4
- This is a QLoRA fine-tune. The original 13b model is a full fine-tune.
It was trained on 1x RTX 6000 Ada for ~17 hours.
- Created
- Jul 6, 2023
- Updated
- Jul 7, 2023