Downloads · 30 days
0
Mini_Orca_7b_v2_2048_Lora_adapter
Vezora/Mini_Orca_7b_v2_2048_Lora_adapter
Mini_Orca_7b_v2_2048_Lora_adapter is a machine learning model from Vezora. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
2 epochs 92 hours of training on a undervolted 3090 pulling 250 watts. My system configuration was getting 13.25 it/s and slowly dropped to 13.04 it/s at 2 epochs. Dataset used is uncencored and filtered of Psmathur's…
Access
Public
Updated Jul 12, 2023
Repo size
67.2 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin67.2 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 12, 2023
- Updated
- Jul 12, 2023
- SHA
- ff8bd7d0
Model card
From the Hugging Face model README
2 epochs 92 hours of training on a undervolted 3090 pulling 250 watts. My system configuration was getting 13.25 it/s and slowly dropped to 13.04 it/s at 2 epochs. Dataset used is uncencored and filtered of Psmathur's wizard orca 72k instructions, ported into a alpaca chat format, and removed all "Input" strings. Trained on only instruction and output. Preforms remarkably well works with the blokes gptq mini orca v2 gptq and allows the model to output 2048 tokens. This adapter was trained to test how lora training could be used to expand context window with less compute. The original model trained by "psmathur" was trained with "8x A100(80G) GPUs for around 13 Hours for cost of $195". I'm hoping the bloke can merge this with the orignal model and upload a gptq version for everyone.
- License
- apache-2.0
- Created
- Jul 12, 2023
- Updated
- Jul 12, 2023