Downloads · 30 days
0
gpt-j-6B-tensorrt-int8
ENOT-AutoDL/gpt-j-6B-tensorrt-int8
gpt-j-6B-tensorrt-int8 is a text generation model from ENOT-AutoDL. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.
Access
Public
Updated Jun 8, 2023
Repo size
49.8 GB
Likes
7
Public
Repo makeup
Click a slice to open those files.
.engine25.5 GB · 51%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Mar 28, 2023
- Updated
- Jun 8, 2023
- SHA
- 2b7d07d4
Try a prompt
Datasets
Model card
From the Hugging Face model README
INT8 GPT-J 6B
GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.
This repository contains GPT-J 6B onnx model suitable for building TensorRT int8+fp32 engines. Quantization of model was performed by the ENOT-AutoDL framework. Code for building of TensorRT engines and examples published on github.
Metrics:
| TensorRT INT8+FP32 | torch FP16 | torch FP32 | |
|---|---|---|---|
| Lambada Acc | 78.46% | 79.53% | - |
| Model size (GB) | 8.5 | 12.1 | 24.2 |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
Latency:
| Input sequance length | Number of generated tokens | TensorRT INT8+FP32 ms | torch FP16 ms | Acceleration |
|---|---|---|---|---|
| 64 | 64 | 1040 | 1610 | 1.55 |
| 64 | 128 | 2089 | 3224 | 1.54 |
| 64 | 256 | 4236 | 6479 | 1.53 |
| 128 | 64 | 1060 | 1619 | 1.53 |
| 128 | 128 | 2120 | 3241 | 1.53 |
| 128 | 256 | 4296 | 6510 | 1.52 |
| 256 | 64 | 1109 | 1640 | 1.49 |
| 256 | 128 | 2204 | 3276 | 1.49 |
| 256 | 256 | 4443 | 6571 | 1.49 |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
How to use
Example of inference and accuracy test published on github:
git clone https://github.com/ENOT-AutoDL/ENOT-transformers
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Languages
- en
- Created
- Mar 28, 2023
- Updated
- Jun 8, 2023