Meta-Llama-3-8B-Instruct-GGUF

SandLogic Technologies - Quantized Meta-Llama3-8b-Instruct Models

Model Description

We have quantized the Meta-Llama3-8b-Instruct model into three variants:

1. Q5_KM
2. Q4_KM
3. IQ4_XS

These quantized models offer improved efficiency while maintaining performance.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

• Name: Meta-Llama3-8b-Instruct
• Developer: Meta
• Release Date: April 18, 2024
• Model Type: Auto-regressive language model
• Architecture: Optimized transformer with Grouped-Query Attention (GQA)
• Parameters: 8 billion
• Context Length: 8k tokens
• Training Data: New mix of publicly available online data (15T+ tokens)
• Knowledge Cutoff: March, 2023

Model Capabilities

Llama 3 is designed for multiple use cases, including:

• Responding to questions in natural language
• Writing code
• Brainstorming ideas
• Content creation
• Summarization

The model understands context and responds in a human-like manner, making it useful for various applications.

Use Cases

1. Chatbots: Enhance customer service automation
2. Content Creation: Generate articles, reports, blogs, and stories
3. Email Communication: Draft emails and maintain consistent brand tone
4. Data Analysis Reports: Summarize findings and create business performance reports
5. Code Generation: Produce code snippets, identify bugs, and provide programming recommendations

Model Variants

We offer three quantized versions of the Meta-Llama3-8b-Instruct model:

1. Q5_KM: 5-bit quantization using the KM method
2. Q4_KM: 4-bit quantization using the KM method
3. IQ4_XS: 4-bit quantization using the IQ4_XS method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Usage

pip install llama-cpp-python 

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/7B/llama-model.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm(
    "Q: Name the planets in the solar system? A: ", # Prompt
    max_tokens=32, # Generate up to 32 tokens
    stop=["Q:", "\n"], # Stop generating just before a new question
    echo=False # Don't echo the prompt in the output
)

print(output["choices"][0]["text"])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Meta-Llama-3-8B-Instruct-GGUF",
    filename="*Meta-Llama-3-8B-Instruct.Q5_K_M.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

License

A custom commercial license is available at: https://llama.meta.com/llama3/license

Acknowledgements

We thank Meta for developing and releasing the original Llama 3 model. Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at [email protected] or visit our support page.