Downloads · 30 days
11
3% of all-time downloads
Mixtral-8x7B-Instruct-v0.1-gpt-fast
kittn/Mixtral-8x7B-Instruct-v0.1-gpt-fast
Mixtral-8x7B-Instruct-v0.1-gpt-fast is a text generation model from kittn. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This repo holds quantized Mixtral-8x7B weights to be used in gpt-fast.
All-time downloads
350
Public
Repo size
46.8 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pth46.8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mixtral
- Access
- Public
- Created
- Dec 19, 2023
- Updated
- Dec 21, 2023
- SHA
- 09a315e7
Model card
From the Hugging Face model README
Mixtral-8x7B (gpt-fast edition)
This repo holds quantized Mixtral-8x7B weights to be used in gpt-fast.
Compatibility
Conversion to int4 was broken, so this repo only holds fp8 weights. Practically speaking this means your GPU(s) need to be Ada Lovelace or newer, and have enough VRAM to hold the model + KV cache + activations.
I'm hoping it can work on a pair of 4090s, which combined have 48 GiB (51.539607552 GB) of VRAM. Ignoring all overhead, this leaves ~4.74 GB for KV-cache and activations, which should be enough (?).
- TODO: Test on 2x4090 with TP=2
Notes
Conversion was done with (commit 7510a9d)
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- apache-2.0
- Languages
- fr, it, de, es, en
- Created
- Dec 19, 2023
- Updated
- Dec 21, 2023