Downloads · 30 days
6
3% of all-time downloads
llama3-ins-8b-int4-trt-llm
muhtasham/llama3-ins-8b-int4-trt-llm
llama3-ins-8b-int4-trt-llm is a machine learning model from muhtasham. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
All-time downloads
190
Public
Repo size
21.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors16.1 GB · 74%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Apr 25, 2024
- Updated
- Apr 25, 2024
- SHA
- 62b019ec
Model card
From the Hugging Face model README
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/muhtasham/llama3-ins-8b-int4-trt-llm
git clone https://github.com/NVIDIA/TensorRT-LLM.git
python ./TensorRT-LLM/examples/run.py --engine_dir=./ \
--max_output_len 5 \
--tokenizer_dir llama3-hf \
--input_text "How do I count to nine in French?" \
--run_profiling
2024-04-25 19:35:59.062455: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Input [Text 0]: "<|begin_of_text|>How do I count to nine in French?"
Output [Text 0 Beam 0]: " Counting in French is"
batch_size: 1, avg latency of 10 iterations: : 0.0999948501586914 sec
- Library
- transformers
- Created
- Apr 25, 2024
- Updated
- Apr 25, 2024