WizardLM-13B-V1.2-int8-ct2

CTranslate2 int8 version of WizardLM-13B-V1.2

This is a int8_float16 quantization of WizardLM-13B-V1.2
See more on CTranslate2: Docs | Github

This model was converted to ct2 format using the following commnd:

ct2-transformers-converter --model WizardLM/WizardLM-13B-V1.2 --copy_files tokenizer.model --output_dir wizard13b --quantization int8_float16 --low_cpu_mem_usage

To convert this model, edits had to be made to the file: added_tokens.json

From:

{
  "<pad>": 32000
}

To:

{
}

no converstion needed using the model from this repository as it is already in ct2 format.

---

From the CTranslate2 GitHub (no relation to this model):

CTranslate2 is a C++ and Python library for efficient inference with Transformer models.

CTranslate2 performance

We translate the En->De test set newstest2014 with multiple models:

• OpenNMT-tf WMT14: a base Transformer trained with OpenNMT-tf on the WMT14 dataset (4.5M lines)
• OpenNMT-py WMT14: a base Transformer trained with OpenNMT-py on the WMT14 dataset (4.5M lines)
• OPUS-MT: a base Transformer trained with Marian on all OPUS data available on 2020-02-26 (81.9M lines)

The benchmark reports the number of target tokens generated per second (higher is better). The results are aggregated over multiple runs. See the benchmark scripts for more details and reproduce these numbers.

Please note that the results presented below are only valid for the configuration used during this benchmark: absolute and relative performance may change with different settings.

CPU

Executed with 4 threads on a c5.2xlarge Amazon EC2 instance equipped with an Intel(R) Xeon(R) Platinum 8275CL CPU.

GPU

Executed with CUDA 11 on a g5.xlarge Amazon EC2 instance equipped with a NVIDIA A10G GPU (driver version: 510.47.03).