Downloads · 30 days
12
14% of all-time downloads
Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit
Aaron2599/Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit
Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit is a machine learning model from Aaron2599. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.1.
- Model creator: Meta-Llama - Original model: Meta-Llama-3.1-8B-Instruct
All-time downloads
85
Public
Repo size
11.5 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin5.7 GB · 100%
At a glance
- License
- llama3.1
- Model type
- llama
- Access
- Public
- Created
- Sep 18, 2024
- Updated
- Sep 18, 2024
- SHA
- aeaf753d
Base models
Model card
From the Hugging Face model README
Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit
- Model creator: Meta-Llama
- Original model: Meta-Llama-3.1-8B-Instruct
Overview
This repository contains a 4-bit AWQ version of Meta-Llama-3.1-8B-Instruct, optimized for the LMDeploy TurboMindEngine. The model is designed to provide efficient and accurate performance with reduced computational requirements.
Model Details
- Model Name: Meta-Llama-3.1-8B-Instruct-TurboMind-AWQ-4bit
- Base Model: meta-llama/Meta-Llama-3.1-8B-Instruct
- Quantization: 4-bit AWQ
- Engine: LMDeploy TurboMindEngine
lmdeploy lite auto_awq \
$HF_MODEL \
--calib-dataset 'ptb' \
--calib-samples 128 \
--calib-seqlen 2048 \
--w-bits 4 \
--w-group-size 128 \
--batch-size 10 \
--search-scale True \
--work-dir $WORK_DIR
- Type
- llama
- License
- llama3.1
- Languages
- en
- Created
- Sep 18, 2024
- Updated
- Sep 18, 2024