Downloads · 30 days
11
1% of all-time downloads
Phi-3-medium-128k-instruct-awq
bjaidi/Phi-3-medium-128k-instruct-awq
Phi-3-medium-128k-instruct-awq is a text generation model from bjaidi. Use it when you need the model to write or continue text. It is set up for transformers.
Generated using autoawq: pip install git+https://github.com/casper-hansen/AutoAWQ.git@f0321eedca887c12680553fc561d176b03b1b9a5 flashattn
All-time downloads
1.8K
Public
Parameters
14B
7.7 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors7.7 GB · 100%
Parameter types
How the weights are stored.
I3213.6B · 98%
Try a prompt
Model card
From the Hugging Face model README
Generated using autoawq: pip install git+https://github.com/casper-hansen/AutoAWQ.git@f0321eedca887c12680553fc561d176b03b1b9a5 flash_attn
Following code used for generation:
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = 'models/Phi-3-medium-128k-instruct'
quant_path = 'models/Phi-3-medium-128k-instruct-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"device_map": "auto"})
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize
model.quantize(tokenizer, quant_config=quant_config)
# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
Original model here: https://huggingface.co/microsoft/Phi-3-medium-128k-instruct
license: mit
- Task
- Text Generation
- Library
- transformers
- Type
- phi3
- Created
- Jun 20, 2024
- Updated
- Jun 20, 2024