bloom-1b7-8bit

<h1 style='text-align: center '>BLOOM LM - 8bit</h1> <h2 style='text-align: center '><em>BigScience Large Open-science Open-access Multilingual Language Model - 8bit</em> </h2> <h3 style='text-align: center '>Model Card</h3> <img src="https://s3.amazonaws.com/moonup/production/uploads/1657124309515-5f17f0a0925b9863e28ad517.png" alt="BigScience Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

Version 1.0 / 26.May.2022

Related paper: https://arxiv.org/abs/2208.07339

TL;DR

This repository contains 8bit weights of bloom-1b7 model. You can load this model using transformers==4.28.0 and bitsandbytes>0.37.2 out of the box !

# pip install accelerate bitsandbytes
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("ybelkada/bloom-1b7-8bit")

How to push 8bit weights?

First, make sure you are using transformers & bitsandbytes versions stated above. Then load your 8bit model as usual using load_in_8bit=True!

# pip install accelerate bitsandbytes
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b7", device_map="auto", load_in_8bit=True)

Then just call push_to_hub method or save_pretrained method if you want to save your 8bit model locally

model.push_to_hub("{your_username}/bloom-1b7-8bit")

That's it!

What is inside the model's state_dict?

Inside the state dict of the model (pytorch_model.bin file) you have

• the quantized int8 weights
• the quantization statistics in float16