Downloads · 30 days
36
21% of all-time downloads
YuvrajSingh9886/facebook-opt-350m-8bit-bnb
facebook-opt-350m-8bit-bnb is a text generation model from YuvrajSingh9886. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This repository contains experiments and implementations of LLM.int8 quantization using BitsAndBytes for OPT (Open Pre-trained Transformer) models. LLM.int8 is a quantization method that converts model weights to 8-bi…
Downloads · 30 days
36
21% of all-time downloads
All-time downloads
175
Public
Parameters
331M
360 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors360 MB · 99%
How the weights are stored.
I8303M · 91%
From the Hugging Face model README
This repository contains experiments and implementations of LLM.int8 quantization using BitsAndBytes for OPT (Open Pre-trained Transformer) models. LLM.int8 is a quantization method that converts model weights to 8-bit precision while maintaining high accuracy through mixed-precision inference.
These models are quantized versions of Facebook's OPT (Open Pre-trained Transformer) models using the LLM.int8 quantization method. The quantization preserves model performance while significantly reducing memory requirements.
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load quantized model
model = AutoModelForCausalLM.from_pretrained("YuvrajSingh9886/facebook-opt-350m-8bit-llm.int8-threshold-8")
tokenizer = AutoTokenizer.from_pretrained("YuvrajSingh9886/facebook-opt-350m-8bit-llm.int8-threshold-8")
# Generate text
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
These quantized models can be used for:
Users should be aware that these models may:
Use the code below to get started with the model.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# For inference with automatic quantization
model = AutoModelForCausalLM.from_pretrained(
"YuvrajSingh9886/facebook-opt-350m-8bit-llm.int8-threshold-8",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("YuvrajSingh9886/facebook-opt-350m-8bit-llm.int8-threshold-8")
# Generate text
text = "The future of AI is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=50, do_sample=True, temperature=0.7)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
The models are quantized versions of pre-trained OPT-350M models. No additional training data was used - quantization is a post-training compression technique.
| Model Variant | Memory (GB) | HellaSwag Acc | Relative Perf |
|---|---|---|---|
| OPT-350M (FP16) | 1.4 | Baseline | 100% |
| LLM.int8 (threshold=6.0) | 0.7 | -0.5% | 99.5% |
| LLM.int8 (threshold=8.0) | 0.7 | -0.3% | 99.7% |
| LLM.int8 (threshold=5.0, lm_head=FP16) | 0.7 | -0.2% | 99.8% |
Note: Performance numbers are approximate and may vary based on evaluation setup
The models maintain the original OPT architecture:
If you use these quantized models in your work, please cite:
@software{smolquant_llm_int8_2024,
title={LLM.int8 Quantized OPT Models},
author={YuvrajSingh-mist},
year={2024},
url={https://github.com/YuvrajSingh-mist/SmolQuant}
}
@inproceedings{zhang2022opt,
title={Opt: Open pre-trained transformer language models},
author={Zhang, Susan and Roller, Stephen and Goyal, Naman and Artetxe, Mikel and Chen, Moya and Chen, Shuohui and Dewan, Christopher and Diab, Mona and Li, Xian and Lin, Xi Victoria and others},
booktitle={arXiv preprint arXiv:2205.01068},
year={2022}
}
@article{dettmers2022llm,
title={LLM. int8 (): 8-bit matrix multiplication for transformers at scale},
author={Dettmers, Tim and Lewis, Mike and Belkada, Younes and Zettlemoyer, Luke},
journal={arXiv preprint arXiv:2208.07339},
year={2022}
}
The repository includes analysis scripts for:
analyze_weights.py - Visualizes weight distributions across layersanalyze_activations.py - Shows activation patterns during inference# Evaluate quantized model
lm_eval --model hf \
--model_args pretrained=YuvrajSingh9886/facebook-opt-350m-8bit-llm.int8-threshold-8 \
--tasks hellaswag \
--device cuda:0 \
--batch_size auto
facebook/opt-350mThis model card follows the HuggingFace model card format and provides comprehensive information about the LLM.int8 quantized OPT models.
<!-- ad-footer -->Released freely — support more experiments like it: