Downloads · 30 days
0
llama-65b-4bit
maderix/llama-65b-4bit
llama-65b-4bit is a machine learning model from maderix. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Converted with https://github.com/qwopqwop200/GPTQ-for-LLaMa All models tested on A100-80G Conversion may require lot of RAM, LLaMA-7b takes ~12 GB, 13b around 21 GB, 30b around 62 and 65b takes more than 120 GB of RAM.
Access
Public
Updated Mar 14, 2023
Repo size
61.3 GB
Likes
66
Public
Repo makeup
Click a slice to open those files.
.pt61.3 GB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Mar 11, 2023
- Updated
- Mar 14, 2023
- SHA
- 7b1be53b
Model card
From the Hugging Face model README
Converted with https://github.com/qwopqwop200/GPTQ-for-LLaMa All models tested on A100-80G *Conversion may require lot of RAM, LLaMA-7b takes ~12 GB, 13b around 21 GB, 30b around 62 and 65b takes more than 120 GB of RAM.
Installation instructions as mentioned in above repo:
- Install Anaconda and create a venv with python 3.8
- Install pytorch(tested with torch-1.13-cu116)
- Install Transformers library (you'll need the latest transformers with this PR : https://github.com/huggingface/transformers/pull/21955 ).
- Install sentencepiece from pip
- Run python cuda_setup.py install in venv
- You can either convert the llama models yourself with the instructions from GPTQ-for-llama repo
- or directly use these weights by individually downloading them following these instructions (https://huggingface.co/docs/huggingface_hub/guides/download)
- Profit!
- Best results are obtained by putting a repetition_penalty(~1/0.85),temperature=0.7 in model.generate() for most LLaMA models
- Library
- transformers
- Languages
- en
- Created
- Mar 11, 2023
- Updated
- Mar 14, 2023