Downloads · 30 days
28
8% of all-time downloads
BigAurelian-v0.5-120b-32k
llmixer/BigAurelian-v0.5-120b-32k
BigAurelian-v0.5-120b-32k is a text generation model from llmixer. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
<img src="https://cdn-uploads.huggingface.co/production/uploads/65a6db055c58475cf9e6def1/QqGLOoVkdN-Z4kLD6OWP6.png" width=600
All-time downloads
341
Public
Parameters
118B
235 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors235 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- llama2
- Model type
- llama
- Access
- Public
- Created
- Jan 16, 2024
- Updated
- Jan 17, 2024
- SHA
- 6eb57d59
Try a prompt
Model card
From the Hugging Face model README
BigAurelian v0.5 120b 32k
<img src="https://cdn-uploads.huggingface.co/production/uploads/65a6db055c58475cf9e6def1/QqGLOoVkdN-Z4kLD6OWP6.png" width=600>A Goliath-120b style frankenmerge of aurelian-v0.5-70b-32K and WinterGoddess-1.4x-70b. The goal is to have similar performance with an extended context size. Important: Use a positional embeddings compression factor (compress_pos_emb) of 8 when loading this model.
Prompting Format
Llama2 and Alpaca.
Merge process
The models used in the merge are aurelian-v0.5-70b-32K and WinterGoddess-1.4x-70b.
The layer mix:
- range 0, 16
aurelian
- range 8, 24
WinterGoddess
- range 17, 32
aurelian
- range 25, 40
WinterGoddess
- range 33, 48
aurelian
- range 41, 56
WinterGoddess
- range 49, 64
aurelian
- range 57, 72
WinterGoddess
- range 65, 80
aurelian
Acknowledgements
@grimulkan For creating aurelian-v0.5-70b-32K
@Sao10K For creating WinterGoddess
@alpindale For creating the original Goliath
@chargoddard For developing mergekit.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama2
- Languages
- en
- Created
- Jan 16, 2024
- Updated
- Jan 17, 2024