Turkish-Llama-3.1-8B-Instruct-AWQ

🇹🇷 Turkish Llama-3.1-8B-Instruct (AWQ)

This repository contains the AWQ (Activation-aware Weight Quantization) formatted version of the Meta-Llama-3.1-8B-Instruct model, which has been fine-tuned on a Turkish instruction dataset.

This AWQ model is highly optimized for blazing-fast inference on GPUs. It reduces the VRAM requirement from ~16 GB down to just ~6 GB, making it incredibly efficient for production-grade serving engines like vLLM while maintaining near FP16 accuracy.

💻 System Requirements

• VRAM: ~6 GB (Can easily run on a single RTX 3060, 4060, or Colab T4)
• Storage: ~6 GB

---

💬 Prompt Template (Llama 3.1 Format)

The Llama 3.1 architecture uses specific <|start_header_id|> and <|eot_id|> tokens. Unlike some other architectures, it natively supports a system role.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Sen yardımsever bir Türkçe asistansın.<|eot_id|><|start_header_id|>user<|end_header_id|>

[Write your prompt here]<|eot_id|><|start_header_id|>assistant<|end_header_id|>

(Note: If you are using the transformers library, tokenizer.apply_chat_template handles this format automatically).

🚀 How to Run the Model

Option 1: Using vLLM (Recommended for High Speed & Production)

vLLM is the fastest way to serve this AWQ model.

1. Install vLLM:

pip install vllm

1. Start an OpenAI-compatible API Server in your terminal:

python -m vllm.entrypoints.openai.api_server \
    --model mehmettozlu/Turkish-Llama-3.1-8B-Instruct-AWQ \
    --quantization awq \
    --max-model-len 4096

1. Query the Server using Python:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

completion = client.chat.completions.create(
    model="mehmettozlu/Turkish-Llama-3.1-8B-Instruct-AWQ",
    messages=[
        {"role": "system", "content": "Sen yardımsever bir Türkçe asistansın."},
        {"role": "user", "content": "Kuantum bilgisayarların çalışma mantığını basitçe açıklar mısın?"}
    ]
)

print(completion.choices[0].message.content)

Option 2: Using Hugging Face transformers (Python)

If you want to run the model directly in a standard Python script (e.g., inside Jupyter or Colab) using GPU acceleration:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mehmettozlu/Turkish-Llama-3.1-8B-Instruct-AWQ"

# Load Tokenizer and Model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)

# Prepare Messages
messages = [
    {"role": "system", "content": "Sen yardımsever bir Türkçe asistansın."},
    {"role": "user", "content": "Python'da nesne yönelimli programlama (OOP) nedir?"}
]

# Apply Llama 3.1 Chat Template automatically
inputs = tokenizer.apply_chat_template(
    messages, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to("cuda")

# Generate Response
outputs = model.generate(
    input_ids=inputs, 
    max_new_tokens=512, 
    temperature=0.3,
    do_sample=True
)

response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)

⚠️ Limitations & Biases

Since this model is based on the Llama 3.1 architecture and fine-tuned on Turkish datasets, it may occasionally hallucinate or reflect biases present in the training data. Users should verify complex factual information and ensure it is not used in high-risk environments without human oversight.