microsoft

bloom-deepspeed-inference-int8

microsoft/bloom-deepspeed-inference-int8

bloom-deepspeed-inference-int8 is a feature extraction model from microsoft. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as bigscience-bloom-rail-1.0.

This is a custom INT8 version of the original BLOOM weights to make it fast to use with the DeepSpeed-Inference engine which uses Tensor Parallelism. In this repo the tensors are split into 8 shards to target 8 GPUs.