Turkish-Mistral-NeMo-12B-Instruct-GGUF

🇹🇷 Turkish Mistral-NeMo-12B-Instruct (GGUF)

This repository contains the GGUF formatted version of the Mistral-Nemo-Instruct-2407 (12B parameters), which has been fine-tuned on a Turkish instruction dataset using Unsloth.

Equipped with the highly efficient Tekken tokenizer and a 128k context window, this model excels at processing Turkish text efficiently. It is optimized to act as a highly capable Turkish AI assistant running locally and 100% offline via llama.cpp, Ollama, and LM Studio.

💾 Available GGUF Files and System Requirements

Because this is a 12-Billion parameter model, it requires slightly more RAM/VRAM than standard 7B/8B models. The q4_k_m version is highly recommended for standard consumer hardware:

---

📊 Model Performance Benchmarks (LLM-as-a-Judge)

This model has been tested under identical conditions alongside other popular Turkish GGUF models and evaluated via an LLM-as-a-Judge benchmark to measure Turkish language proficiency, instruction-following capabilities, and coding performance.

🏆 Comparison Table

💬 Prompt Template (Mistral Instruct Format)

To get the best performance and prevent hallucinations, you must use the standard Mistral Instruct structure ([INST] and [/INST] tokens). System prompts can be prepended inside the initial instruction block:

<s>[INST] Sen yardımsever bir Türkçe asistansın.

[Write your prompt here] [/INST]

🚀 How to Run the Model

You can run this model locally with complete privacy and zero internet connection required.

Option 1: Using Python (llama-cpp-python)

1. Install the library via pip:

pip install llama-cpp-python

1. Download the model from Hugging Face:

wget -O Turkish-Mistral-NeMo-12B-Instruct-q4_k_m.gguf [https://huggingface.co/mehmettozlu/Turkish-Mistral-NeMo-12B-Instruct-GGUF/resolve/main/Turkish-Mistral-NeMo-12B-Instruct-q4_k_m.gguf](https://huggingface.co/mehmettozlu/Turkish-Mistral-NeMo-12B-Instruct-GGUF/resolve/main/Turkish-Mistral-NeMo-12B-Instruct-q4_k_m.gguf)

1. Create a Python script (run.py):

from llama_cpp import Llama

llm = Llama(
    model_path="./Turkish-Mistral-NeMo-12B-Instruct-q4_k_m.gguf",
    n_ctx=4096,          # Context window size (can be increased up to 128k based on your RAM)
    n_gpu_layers=-1      # Offload all layers to GPU
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "Sen yardımsever bir Türkçe asistansın."},
        {"role": "user", "content": "Yapay zeka modellerinin nasıl eğitildiğini kısaca açıklar mısın?"}
    ]
)

print(response["choices"][0]["message"]["content"])

Option 2: Using LM Studio (Easiest - GUI)

• Download and install LM Studio.
• In the search bar, type mehmettozlu/Turkish-Mistral-NeMo-12B-Instruct-GGUF.
• Download the q4_k_m.gguf file.
• Navigate to the "Chat" tab, ensure the prompt format is set to "Mistral Instruct", and start chatting in Turkish.

Option 3: Using Ollama (For Developers)

• Create a text file named Modelfile (no extension) and paste the following content:

FROM hf.co/mehmettozlu/Turkish-Mistral-NeMo-12B-Instruct-GGUF/Turkish-Mistral-NeMo-12B-Instruct-q4_k_m.gguf

TEMPLATE """{{ if .System }}<s>[INST] {{ .System }}

{{ end }}{{ .Prompt }} [/INST]"""
SYSTEM """Sen yardımsever bir Türkçe asistansın."""
PARAMETER stop "</s>"
PARAMETER stop "[INST]"
PARAMETER stop "[/INST]"

• Build and run the model:

ollama create turkish-nemo-12b -f Modelfile
ollama run turkish-nemo-12b