Downloads · 30 days
0
0% of all-time downloads
MoE-Mixtral-7B-8Expert
Alpha-VLLM/MoE-Mixtral-7B-8Expert
MoE-Mixtral-7B-8Expert is a machine learning model from Alpha-VLLM. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
mixtral-8x7b is a Mixture-of-Expert (MoE) model. LLaMA2-Accessory has supported its inference and finetuning.
All-time downloads
2
Public
Repo size
520 GB
Likes
17
Public
Repo makeup
Click a slice to open those files.
.pth426 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Dec 10, 2023
- Updated
- Dec 15, 2023
- SHA
- c7224ec0
Model card
From the Hugging Face model README
🔥 MoE-Mixtral-7B-8Expert
mixtral-8x7b is a Mixture-of-Expert (MoE) model. LLaMA2-Accessory has supported its inference and finetuning.
🚀 Features
With LLaMA2-Accessory, mixtral-8x7b enjoys the following features:
- Distributed MoE (namely instantiating experts on multiple processes/gpus)
- Load Balancing Loss
- Tensor Parallel and FSDP for efficiently training
- Distributed and/or quantized inference
🔥 Online Demo
We host a web demo 💻here, which shows a mixtral-8x7b model finetuned on evol-codealpaca-v1 and ultrachat_200k, with LoRA and Bias tuning.
💡 Tutorial
A detailed tutorial is available at our document
- License
- mit
- Created
- Dec 10, 2023
- Updated
- Dec 15, 2023