llama-3.1-8b-instruct-gguf

Llama 3.1 8B Instruct - GGUF (Optimized)

This repository contains an optimized GGUF quantized version of Llama 3.1 8B Instruct for fast inference with llama.cpp.

Model Details

• Base Model: Meta-Llama-3.1-8B-Instruct
• Quantization: Q4_K_M (4-bit, k-quant, medium)
• File Size: ~4.9GB
• Use Case: Balanced performance and quality

Quick Start

Option 1: Download and Use with llama.cpp

# Download the model
huggingface-cli download Khushiee/llama-3.1-8b-instruct-gguf Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir ./models

# Run inference
./llama-cli -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p "Hello, how are you?"

Option 2: Use with Python (llama-cpp-python)

pip install llama-cpp-python

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
    n_ctx=4096,  # Context window
    n_threads=8,  # CPU threads
    n_gpu_layers=35  # Use GPU if available (0 for CPU only)
)

# Generate text
output = llm(
    "Write a short poem about AI:",
    max_tokens=256,
    temperature=0.7
)

print(output['choices'][0]['text'])

Option 3: Use with LangChain

from langchain_community.llms import LlamaCpp

llm = LlamaCpp(
    model_path="./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    temperature=0.1
)

response = llm.invoke("What is machine learning?")
print(response)

Performance Tips

• CPU Only: Use Q4_K_M (this file), set n_gpu_layers=0
• With GPU: Set n_gpu_layers=35 or higher for all layers
• Faster inference: Increase n_batch to 512 or 1024
• More threads: Set n_threads to your CPU core count

System Requirements

• RAM: 8GB minimum (16GB recommended)
• Storage: 5GB free space
• CPU: Modern x86_64 or ARM processor
• GPU (optional): CUDA or Metal support for acceleration

Model Info

This is a quantized version optimized for:

• ✅ Fast inference speed
• ✅ Low memory usage
• ✅ Good quality output
• ✅ Tool/function calling support

License

This model follows the original Llama 3.1 license from Meta.

Credits

• Original model: Meta
• GGUF conversion: bartowski
• Uploaded by: Khushiee