Downloads · 30 days
8
11% of all-time downloads
Llama-3-8B-Instruct-EXL3-4.0bpw
FerrisIntelligence/Llama-3-8B-Instruct-EXL3-4.0bpw
Llama-3-8B-Instruct-EXL3-4.0bpw is a text generation model from FerrisIntelligence. Use it when you need the model to write or continue text. The card lists the license as llama3.
This repository contains a 4.0bpw EXL3 quantization of the Meta-Llama-3-8B-Instruct model, optimized specifically for high-speed inference on NVIDIA RTX GPUs.
All-time downloads
70
Public
Parameters
2.5B
4.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.9 GB · 100%
Parameter types
How the weights are stored.
I161.9B · 79%
Try a prompt
Base models
Model card
From the Hugging Face model README
Llama-3-8B-Instruct-EXL3-4.0bpw
This repository contains a 4.0bpw EXL3 quantization of the Meta-Llama-3-8B-Instruct model, optimized specifically for high-speed inference on NVIDIA RTX GPUs.
Why EXL3?
EXL3 (part of the ExLlamaV2 ecosystem) offers the best performance-to-vram ratio for NVIDIA users. On an RTX 5070 (12GB), this model provides lightning-fast token generation while leaving enough VRAM headroom for large context windows or background tasks.
Model Highlights
- Format: EXL3 (ExLlamaV2)
- Precision: 4.0 bits per weight (bpw)
- Architecture: Llama 3
- VRAM Friendly: Fits comfortably in 8GB and 12GB cards.
VRAM Usage Estimates
| Context Length | VRAM Usage (Approx) |
|---|---|
| 4,096 tokens | ~5.3 GB |
| 8,192 tokens | ~6.2 GB |
| 16,384 tokens | ~7.8 GB |
How to use
This model is designed for use with ExLlamaV2 loaders.
Local Inference (TabbyAPI / Oobabooga)
- Point your loader to this repository folder.
- Select the
ExLlamaV2loader. - Ensure you have the latest CUDA drivers (12.x recommended).
Integration (Python)
from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Tokenizer
config = ExLlamaV2Config()
config.model_dir = "path/to/Llama-3-8B-Instruct-EXL3-4.0bpw"
config.prepare()
model = ExLlamaV2(config)
model.load()
- Task
- Text Generation
- Type
- llama
- License
- llama3
- Languages
- en
- Created
- Jan 21, 2026
- Updated
- Feb 7, 2026