AhiskaAI-135m-Base-v0.3

AhıskaAI-135M-Base-v0.3

AhıskaAI-135M-Base-v0.3 is a ~135 million parameter Small Language Model (SLM) trained completely from scratch. Building upon the learnings of the v0.2 release, v0.3 introduces architectural optimizations (such as Grouped-Query Attention), an expanded and multi-domain pre-training corpus, and enhanced training efficiency using Liger Kernel.

Key Improvements from v0.2 to v0.3

• Architectural Evolution (GQA Support): Transitioned to Grouped-Query Attention (GQA) with 12 query heads and 4 key-value heads (num_key_value_heads: 4). This significantly reduces KV-cache memory overhead and boosts inference throughput compared to standard MHA in v0.2.
• Diversified Pre-training Mixture: Expanded beyond single-source corpora to a multi-domain dataset totaling ~2.04 GB across Turkish Wikipedia (%53.2), Turkish Stories (%21.6), Python Code (%17.6), and English FineWeb (%7.6).
• Training Acceleration & Efficiency: Utilized Liger Kernel integration (apply_liger_kernel_to_llama) and adamw_torch_fused optimization in bfloat16 precision, maximizing hardware utilization.
• Hardware & Scale: Scaled pre-training across dual Kaggle NVIDIA T4 GPUs for 11 hours, overcoming consumer laptop VRAM limitations while preserving tight parameter footprints.

Model Details

• Architecture: LlamaForCausalLM with GQA
• Parameters: ~135M
• Hidden Size: 768
• Intermediate Size: 3072
• Attention Heads: 12 (Query) / 4 (Key/Value)
• Layers: 14
• Context Length: 512 tokens
• Vocabulary Size: 32,000
• Precision: float16

Pre-training Dataset Distribution

The model was pre-trained on a balanced multi-domain dataset (~2.04 GB total):

480M Tokens

Training Hardware & Setup

• Hardware: 2x NVIDIA T4 GPUs (Kaggle)
• Training Time: ~11 Hours
• Optimization: SDPA (Scaled Dot-Product Attention), AdamW Fused, Liger Kernel
• Precision: float16

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhiskaAI/AhiskaAI-135M-Base-v0.3"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto"
)

prompt = "Türkiye Cumhuriyeti'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Related Models

• Instruct Version: AhiskaAI/AhiskaAI-135M-IT-v0.3 (SFT ChatML model fine-tuned for instruction following)

About AhıskaAI

AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.