AhiskaAI-10M-Experimental-v0.1-Base

AhıskaAI-10M-Experimental-v0.1

AhıskaAI-10M-Experimental-v0.1 is an ultra-compact ~10 million parameter Small Language Model (SLM) trained from scratch. As part of the AhıskaAI experimental series, this micro-model is designed to test the absolute lower limits of Turkish language representations, syntax learning, and synthetic task understanding at minimal computational costs.

Experimental Scope & Setup

• Objective: Evaluate how much linguistic structure and reasoning a 10M parameter micro-model can capture when trained on dense synthetic data.
• Training Epochs: Pre-trained for 1 full epoch across a ~3.5 GB synthetic Turkish corpus.
• Vocabulary Efficiency: Uses an optimized 8,000 token vocabulary specifically suited for micro-scale models.

Model Details

• Architecture: LlamaForCausalLM (Micro MHA)
• Parameters: ~10M
• Hidden Size: 288
• Intermediate Size: 768
• Attention Heads: 8 (Query) / 8 (Key/Value)
• Layers: 8
• Context Length: 512 tokens
• Vocabulary Size: 8,000
• Precision: float32

Pre-training Dataset Mixture

The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhıskaAI/AhıskaAI-10M-Experimental-v0.1-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

prompt = "Türkiye"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

About AhıskaAI

AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.