Downloads · 30 days
0
Meta-Llama-3-70B-Instruct-Distributed
Azamorn/Meta-Llama-3-70B-Instruct-Distributed
Meta-Llama-3-70B-Instruct-Distributed is a machine learning model from Azamorn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.
Quantized Llama 3 70B Instruct to Q40 format supported by Distributed Llama.
Access
Public
Updated May 27, 2024
Repo size
43.3 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.m43.3 GB · 100%
At a glance
- License
- llama3
- Access
- Public
- Created
- May 27, 2024
- Updated
- May 27, 2024
- SHA
- 0d2a2fb8
Model card
From the Hugging Face model README
Quantized Llama 3 70B Instruct to Q40 format supported by Distributed Llama.
License
Before download this repository please accept Llama 3 Community License.
How to run
- Clone this repository.
- Clone Distributed Llama:
git clone https://github.com/b4rtaz/distributed-llama.git
- Build Distributed Llama:
make dllama
- Run Distributed Llama:
sudo nice -n -20 ./dllama inference --model /path/to/dllama_model_llama3-70b-instruct_q40.m --tokenizer /path/to/dllama_tokenizer_llama3.t --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 4
Chat Template
Please keep in mind this model expects the prompt to use the chat template of llama 3.
- License
- llama3
- Created
- May 27, 2024
- Updated
- May 27, 2024