Downloads · 30 days
15
6% of all-time downloads
internlm2-chat-20b-nf4
samgreen/internlm2-chat-20b-nf4
internlm2-chat-20b-nf4 is a feature extraction model from samgreen. Use it when you need embeddings to search or compare text. It is set up for transformers.
As of 2024/1/17, Transformers must be installed from source and bitsandbytes =0.42.0 is required in order to load serialized 4-bit quants.
All-time downloads
245
Public
Parameters
20.4B
11.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors11.9 GB · 100%
Parameter types
How the weights are stored.
U819.3B · 94%
Model card
From the Hugging Face model README
InterLM2-Chat NF4 Quant
Usage
As of 2024/1/17, Transformers must be installed from source and bitsandbytes >=0.42.0 is required in order to load serialized 4-bit quants.
pip install -U git+https://github.com/huggingface/transformers bitsandbytes
Quantization config
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
Not necessary for inference, just load the model without specifying any quantization/load_in_*bit.
Model Details
- Task
- Feature Extraction
- Library
- transformers
- Type
- internlm
- Created
- Jan 17, 2024
- Updated
- Jan 17, 2024