dolly-v2-12b-sharded-8bit

dolly-v2-12b: sharded 8bit checkpoint

<a href="https://colab.research.google.com/gist/pszemraj/1bc9cea67e6c8dc450b868e0cfc18163/dolly-v2-12b-8bit-inference.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>

This is a sharded checkpoint (with ~4GB shards) of the databricks/dolly-v2-12b model in 8bit precision using bitsandbytes.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the 4.28.0 release info.

• total model size is only ~12.5 GB!
• this enables low-RAM loading, i.e. Colab :)
• update: generation speed can be greatly improved by setting use_cache=True and generating via contrastive search. example notenook here

Basic Usage

install/upgrade transformers, accelerate, and bitsandbytes. For this to work you must have transformers>=4.28.0 and bitsandbytes>0.37.2.

pip install -U -q transformers bitsandbytes accelerate

Load the model. As it is serialized in 8bit you don't need to do anything special:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-12b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)