Mellum2-12B-A2.5B-Instruct-GGUF

Mellum2-12B-A2.5B-Instruct-GGUF

This repository contains GGUF quantizations of JetBrains/Mellum2-12B-A2.5B-Instruct for efficient local inference with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

Model details

• Base model: JetBrains/Mellum2-12B-A2.5B-Instruct
• Format: GGUF
• Architecture: Mellum2
• Task type: Instruction-following assistant
• Context length: 131,072 tokens
• License: Apache 2.0

Available quantizations

Intended use

This model is intended for:

• local chat and assistant workflows
• coding assistance
• tool-use experiments
• CPU-friendly or memory-constrained inference
• users who want a balance between quality and memory usage

Notes

These quantizations were produced with llama.cpp. During conversion, some tensors may require fallback quantization depending on the model architecture, which is expected and does not prevent successful inference.

Usage

llama.cpp

./llama-server -m Mellum2-12B-A2.5B-Instruct-Q5_K_M.gguf --jinja --port 8000

Ollama

Create a Modelfile pointing to the GGUF file:

FROM ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf

Then run:

ollama create mellum2-q4 -f Modelfile
ollama run mellum2-q4

License

Released under the same license as the base model: Apache 2.0.

Acknowledgments

• Base model: JetBrains/Mellum2-12B-A2.5B-Instruct
• Quantization performed with llama.cpp