Gemma-2-9b-it-GGUF

SandLogic Technologies - Quantized Gemma-2-9b-IT Models

Model Description

We have quantized the Gemma-2-9b-IT model into three variants:

1. Q5_KM
2. Q4_KM
3. IQ4_XS

These quantized models offer improved efficiency while maintaining performance.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

• Name: Gemma-2-9b-IT
• Developer: Google
• Model Type: Text-to-text, decoder-only large language model
• Architecture: Based on Gemini technology
• Parameters: 9 billion
• Training Data: 8 trillion tokens, including web documents, code, and mathematics
• Language: English

Model Capabilities

Gemma is designed for various text generation tasks, including:

• Question answering
• Summarization
• Reasoning
• Creative writing
• Code generation

The model is lightweight and suitable for deployment in resource-limited environments such as laptops, desktops, or personal cloud infrastructure.

Use Cases

1. Text Generation: Create poems, scripts, code, marketing copy, and email drafts
2. Chatbots and Conversational AI: Power customer service interfaces, virtual assistants, and interactive applications
3. Text Summarization: Generate concise summaries of text corpora, research papers, or reports

Model Variants

We offer three quantized versions of the Gemma-2-9b-IT model:

1. Q5_KM: 5-bit quantization using the KM method
2. Q4_KM: 4-bit quantization using the KM method
3. IQ4_XS: 4-bit quantization using the IQ4_XS method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Usage

pip install llama-cpp-python 

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/7B/llama-model.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm(
    "Q: Name the planets in the solar system? A: ", # Prompt
    max_tokens=32, # Generate up to 32 tokens
    stop=["Q:", "\n"], # Stop generating just before a new question
    echo=False # Don't echo the prompt in the output
)

print(output["choices"][0]["text"])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Gemma-2-9b-it-GGUF",
    filename="*gemma-2-9b-it-IQ4_XS.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

Input and Output

• Input: Text string (e.g., question, prompt, or document to be summarized)
• Output: Generated English-language text in response to the input

License

Gemma 2 License: Google gemma

Acknowledgements

We thank Google for developing and releasing the original Gemma model. Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at [email protected] or visit our support page.