Downloads · 30 days
19
21% of all-time downloads
TowerInstruct-7B-v0.1-FP8
muhtasham/TowerInstruct-7B-v0.1-FP8
TowerInstruct-7B-v0.1-FP8 is a text generation model from muhtasham. Use it when you need the model to write or continue text. It is set up for transformers.
All-time downloads
89
Public
Parameters
6.7B
7.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors7.5 GB · 100%
Parameter types
How the weights are stored.
F8_E4M36.5B · 96%
Try a prompt
Model card
From the Hugging Face model README
git clone https://github.com/neuralmagic/AutoFP8.git
pip install -e AutoFP8
from transformers import AutoTokenizer
from auto_fp8 import AutoFP8ForCausalLM, BaseQuantizeConfig
from datasets import load_dataset
pretrained_model_dir = "Unbabel/TowerInstruct-7B-v0.1"
quantized_model_dir = "TowerInstruct-7B-v0.1-FP8"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token
ds = load_dataset("mgoin/ultrachat_2k", split="train_sft").select(range(512))
examples = [tokenizer.apply_chat_template(batch["messages"], tokenize=False) for batch in ds]
examples = tokenizer(examples, padding=True, truncation=True, return_tensors="pt").to("cuda")
quantize_config = BaseQuantizeConfig(quant_method="fp8", activation_scheme="static")
model = AutoFP8ForCausalLM.from_pretrained(
pretrained_model_dir, quantize_config=quantize_config
)
model.quantize(examples)
model.save_quantized(quantized_model_dir)
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Jul 11, 2024
- Updated
- Aug 27, 2024