Downloads · 30 days
14.8K
23% of all-time downloads
ytu-ce-cosmos/Turkish-Llama-8b-Instruct-v0.1-GGUF
Turkish-Llama-8b-Instruct-v0.1-GGUF is a machine learning model from ytu-ce-cosmos. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.
Due to the need for quantized models in real-time applications, we introduce our GGUF formatted models. These models are part of GGML project with a hope to democratize the use of Large Models. Depending on the quanti…
Downloads · 30 days
14.8K
23% of all-time downloads
All-time downloads
64.9K
Public
Repo size
100 GB
Likes
27
Public
Click a slice to open those files.
.gguf115 GB · 100%
From the Hugging Face model README
Due to the need for quantized models in real-time applications, we introduce our GGUF formatted models. These models are part of GGML project with a hope to democratize the use of Large Models. Depending on the quantization type, there are 20+ models.
llama.cpp environments, llama-cli and llama-server.lmstudio to utilize these models. 👇

Usage example with llama-cpp-python
from llama_cpp import Llama
# Define the inference parameters
inference_params = {
"n_threads": 4,
"n_predict": -1,
"top_k": 40,
"min_p": 0.05,
"top_p": 0.95,
"temp": 0.8,
"repeat_penalty": 1.1,
"input_prefix": "<|start_header_id|>user<|end_header_id|>\\n\\n",
"input_suffix": "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n",
"antiprompt": [],
"pre_prompt": "Sen bir yapay zeka asistanısın. Kullanıcı sana bir görev verecek. Amacın görevi olabildiğince sadık bir şekilde tamamlamak.",
"pre_prompt_suffix": "<|eot_id|>",
"pre_prompt_prefix": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\n",
"seed": -1,
"tfs_z": 1,
"typical_p": 1,
"repeat_last_n": 64,
"frequency_penalty": 0,
"presence_penalty": 0,
"n_keep": 0,
"logit_bias": {},
"mirostat": 0,
"mirostat_tau": 5,
"mirostat_eta": 0.1,
"memory_f16": True,
"multiline_input": False,
"penalize_nl": True
}
# Initialize the Llama model with the specified inference parameters
llama = Llama.from_pretrained(
repo_id="ytu-ce-cosmos/Turkish-Llama-8b-Instruct-v0.1-GGUF",
filename="*Q4_K.gguf",
verbose=False
)
# Example input
user_input = "Türkiyenin başkenti neresidir?"
# Construct the prompt
prompt = f"{inference_params['pre_prompt_prefix']}{inference_params['pre_prompt']}\n\n{inference_params['input_prefix']}{user_input}{inference_params['input_suffix']}"
# Generate the response
response = llama(prompt)
# Output the response
print(response['choices'][0]['text'])
The quantization has been made using llama.cpp. As we have seen, this method tends to give the most stable results.
Obviously, we encountered better inference quality for models with the highest bits. However, the inference time tends to be similar between low-bit models.
Each model's memory footprint can be anticipated by the qunatization docs in either Hugging Face or llama.cpp.
@inproceedings{kesgin2024optimizing,
title={Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training},
author={Kesgin, H Toprak and Yuce, M Kaan and Dogan, Eren and Uzun, M Egemen and Uz, Atahan and {\.I}nce, Elif and Erdem, Yusuf and Shbib, Osama and Zeer, Ahmed and Amasyali, M Fatih},
booktitle={2024 Innovations in Intelligent Systems and Applications Conference (ASYU)},
pages={1--6},
year={2024},
organization={IEEE}
}
COSMOS AI Research Group, Yildiz Technical University Computer Engineering Department
https://cosmos.yildiz.edu.tr/
[email protected]