Downloads · 30 days
14
3% of all-time downloads
Phi3-128k-Instruct-4Bit-GPTQ
Granther/Phi3-128k-Instruct-4Bit-GPTQ
Phi3-128k-Instruct-4Bit-GPTQ is a text generation model from Granther. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
- 4 Bit Quantized version of Microsoft's Phi3 Mini 128k: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct - Quantized the model with HuggingFace's 🤗 GPTQQuanizer
All-time downloads
537
Public
Parameters
3.8B
2.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.3 GB · 100%
Parameter types
How the weights are stored.
I323.6B · 95%
Try a prompt
Model card
From the Hugging Face model README
Phi3 Mini 128k 4 Bit Quantized
- 4 Bit Quantized version of Microsoft's Phi3 Mini 128k: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
- Quantized the model with HuggingFace's 🤗 GPTQQuanizer
Flash Attention
- The Phi3 family supports Flash Attenion 2, this mechanism allows for faster inference with lower resource use.
- When quantizing Phi3 on a 4090 (24G) with Flash Attention disabled Quantization would fail due to insufficient VRAM
- Enabling Flash Attention allowed Quantization to complete with an extra 10 Giagbaytes of VRAM available on the GPU
Metrics
Total Size:
- Before: 7.64G
- After: 2.28G
VRAM Size:
- Before: 11.47G
- After: 6.57G
Average Inference Time:
- Before: 12ms/token
- After: 5ms/token
- Task
- Text Generation
- Library
- transformers
- Type
- phi3
- License
- mit
- Languages
- en
- Created
- Jun 28, 2024
- Updated
- Jul 1, 2024