Downloads · 30 days
11
19% of all-time downloads
Meta-Llama-3-70B-Instruct-AutoAWQ-4bit
Melon/Meta-Llama-3-70B-Instruct-AutoAWQ-4bit
Meta-Llama-3-70B-Instruct-AutoAWQ-4bit is a text generation model from Melon. Use it when you need the model to write or continue text. It is set up for transformers.
You can deploy it using vllm. And this is the script for deploying.
All-time downloads
59
Public
Parameters
70.6B
39.8 GB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.safetensors39.8 GB · 100%
Parameter types
How the weights are stored.
I3268.5B · 97%
Try a prompt
Model card
From the Hugging Face model README
You can deploy it using vllm. And this is the script for deploying.
python -O -u -m vllm.entrypoints.openai.api_server \
--host=127.0.0.1 \
--port=8090 \
--model=Melon/Meta-Llama-3-70B-Instruct-AutoAWQ-4bit \
--tokenizer=meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size=1 \
--quantization awq \
--dtype half
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Apr 25, 2024
- Updated
- Apr 29, 2024