AhiskaAI-110M-Experimental-v0.1-Instruct

AhıskaAI-110M-IT-Experimental-v0.1

AhıskaAI-110M-IT-Experimental-v0.1 is an instruction-tuned Small Language Model (~110M parameters) fine-tuned from AhıskaAI-110M-Experimental-v0.1. It is designed to evaluate early instruction-following capabilities on top of synthetic pre-trained representations, using multi-turn Turkish conversations with the ChatML template.

Model Highlights

• Instruction Alignment: Supervised Fine-Tuned (SFT) using custom-cleaned multi-turn instruction datasets formatted in ChatML.
• Loss Masking Strategy: Trained using custom prompt masking (labels = -100 for user/system tokens), ensuring loss is calculated only on assistant responses for concise and non-hallucinating outputs.
• System Prompt Support: Native support for fixed system prompts prioritizing polite, short, and accurate Turkish answers.
• Hardware-Efficient Fine-Tuning: Trained on consumer-grade hardware (NVIDIA RTX 4050 6GB GPU) using Liger Kernel acceleration (apply_liger_kernel_to_llama) and adamw_torch_fused.

Model Details

• Base Model: AhıskaAI/AhıskaAI-110M-Experimental-v0.1
• Architecture: LlamaForCausalLM (MHA)
• Parameters: ~110M
• Fine-Tuning Method: Full Parameter SFT (Supervised Fine-Tuning)
• Context Length: 1024 tokens
• Template: ChatML (<|im_start|> and <|im_end|>)
• Precision: Bf16

Supported System Prompts

The model has been optimized around two primary system personas:

1. "Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."
2. "Sen AhıskaAI adında Türkçe bir yapay zeka asistansın.\nGörevlerin:\n1. Sorulara doğrudan, net ve kısa cümlelerle cevap ver.\n2. Bilmediğin veya emin olmadığın konularda uydurma yapma, bilmiyorum de.\n3. Kullanıcının verdiği metin veya listeleri istenen formata sadık kalarak düzenle."

Usage (with Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhıskaAI/AhıskaAI-110M-IT-Experimental-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

# ChatML Formatting
system_prompt = "Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "11 sayısından 2 çıkarırsak kaç kalır? Açıkla."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=0.3,
    top_p=0.9,
    do_sample=True
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Training Parameters & Hardware

Epochs: 1 Learning Rate: 5e-5 (Cosine Schedule) Batch Size: 32 (effective) Optimizer: AdamW Fused (adamw_torch_fused) Acceleration: Liger Kernel (liger-kernel) & SDPA Hardware: NVIDIA RTX 4050 Laptop GPU (6GB VRAM)

Related Resources

Base Model: AhıskaAI-110M-Experimental-v0.1

About AhıskaAI

AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.