dolly-v2-7b-sharded-8bit

dolly-v2-7b: 8-bit sharded checkpoint

<a href="https://colab.research.google.com/gist/pszemraj/8100e98caab538be32832d1208e93f65/dolly-v2-7b-8bit-inference.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>

This is a sharded checkpoint (with ~2GB shards) of the databricks/dolly-v2-7b model in 8-bit precision using bitsandbytes.

Refer to the original model for all details. For more info on loading 8bit models, refer to the example repo and/or the 4.28.0 release info.

• total model size is only ~7.5 GB!
• this enables low-RAM loading, i.e. Colab :)

Basic Usage

install/upgrade transformers, accelerate, and bitsandbytes. For this to work you must have transformers>=4.28.0 and bitsandbytes>0.37.2.

pip install -U -q transformers bitsandbytes accelerate

Load the model. As it is serialized in 8bit you don't need to do anything special:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-7b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)