Downloads · 30 days
3
13% of all-time downloads
mistruct3-trtllm-awq-a4000
dgtdgt/mistruct3-trtllm-awq-a4000
mistruct3-trtllm-awq-a4000 is a machine learning model from dgtdgt. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
f430a4b447ef4cba22698902d43eae0debf08594
All-time downloads
24
Public
Repo size
8.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.engine4.2 GB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Jun 3, 2024
- Updated
- Jun 12, 2024
- SHA
- 0669067a
Model card
From the Hugging Face model README
f430a4b447ef4cba22698902d43eae0debf08594
python ../quantization/quantize.py --model_dir /Mistral-7B-Instruct-v0.3
--dtype float16
--qformat int4_awq
--awq_block_size 128
--output_dir ./quantized_int4-awq
--calib_size 32
trtllm-build --checkpoint_dir /mistruct3trtllm/quantized-i4awq --output_dir ./awq_engine --gemm_plugin auto --max_batch_size 32 --max_input_len 8192 --max_output_len 4096 --max_beam_width 1 --max_num_tokens 16384
python3 ../run.py --engine_dir /workspaces/models/awq_engine --max_output_len 100 --tokenizer_dir mistralai/Mistral-7B-Instruct-v0.3 --input_text "How do I count to nine in French?"
- Library
- transformers
- Created
- Jun 3, 2024
- Updated
- Jun 12, 2024