Downloads · 30 days
0
Meta-Llama-3-70B-Instruct-q4f16_1-MLC
bayley/Meta-Llama-3-70B-Instruct-q4f16_1-MLC
Meta-Llama-3-70B-Instruct-q4f16_1-MLC is a machine learning model from bayley. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.
Llama 3 70B Instruct quantized for use with MLC-LLM
Access
Public
Updated May 9, 2024
Repo size
39.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin39.7 GB · 100%
At a glance
- License
- llama3
- Access
- Public
- Created
- May 9, 2024
- Updated
- May 9, 2024
- SHA
- 5038c182
Model card
From the Hugging Face model README
Llama 3 70B Instruct quantized for use with MLC-LLM
MLC-LLM is an Apache TVM based inference framework with a neat trick: out of all of the frameworks that support tensor parallel inference, MLC-LLM is by far the easiest to install for single-user inference. This allows for linear-ish performance scaling on 2x 3090, 2x 4090, or 2x 7900 XTX, achieving about 30 tokens per second. In particular, this is faster than the performance from a single 48GB card costing much more.
MLC-LLM requires Ubuntu 22.04 or above (the prebuilt wheels will not work on Ubuntu 20.04). For CUDA users:
python3 -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu122 mlc-ai-nightly-cu122
git clone https://huggingface.co/bayley/Meta-Llama-3-70B-Instruct-q4f16_1-MLC
mlc_llm compile Meta-Llama-3-70B-Instruct-q4f16_1-MLC/mlc-chat-config.json --device cuda --overrides "tensor_parallel_shards=<number of gpus>" -o Meta-Llama-3-70B-Instruct-q4f16_1-cuda.so
mlc_llm serve Meta-Llama-3-70B-Instruct-q4f16_1-MLC --model-lib Meta-Llama-3-70B-Instruct-q4f16_1-cuda.so --host 0.0.0.0
This should start an OpenAI-compatible REST server serving a chat completions endpoint that you can connect to with your favorite frontend.
- License
- llama3
- Created
- May 9, 2024
- Updated
- May 9, 2024