Kara-Kumru-v1.0-2B

Kara-Kumru-v1.0-2B 🐦‍⬛

A 2B parameter Turkish LLM that outperforms 70B models on Turkish benchmarks.

Kara-Kumru-v1.0-2B is a fine-tuned version of vngrs-ai/Kumru-2B, specifically optimized for Turkish language tasks including question answering, summarization, and translation. Despite having only 2 billion parameters, it achieves 37.56 average on the Cetvel Turkish LLM Benchmark, surpassing Llama-3.3-70B-Instruct (36.25) — a model 35x its size.

<p align="center"> <img src="kara_kumru_v1_benchmark_v2.png" alt="Cetvel Turkish LLM Benchmark Leaderboard" width="100%"> </p>

<p align="center">Leaderboard scores for other models are sourced from vngrs-ai/Kumru-2B. Kara-Kumru-v1.0-2B scores were evaluated using our own Cetvel pipeline.</p>

Key Results

🥇 Kara-Kumru-v1.0-2B achieves the highest QA and SUM scores across the entire Cetvel leaderboard, including models up to 72B parameters.

Detailed Task-Level Results

<details> <summary>Click to expand full task breakdown</summary> </details>

Cetvel Leaderboard Position

#1  Kumru-7B                   41.58  (7B)
#2  Kara-Kumru-v1.0-2B            37.56  (2B) ← YOU ARE HERE
#3  Llama-3.3-70B-Instruct     36.25  (70B)
#4  Kumru-2B                   31.98  (2B)
#5  gemma-3-27b-it             27.73  (27B)
#6  gemma-3-12b-it             27.60  (12B)
#7  Qwen2-72B-Instruct         26.07  (72B)
    ...

Highlights

• 35x smaller, higher score: 2B params beating Llama-3.3-70B-Instruct on Turkish
• Best-in-class QA: 32.54 — highest QA score across ALL models in the Cetvel leaderboard, including 72B models
• Best-in-class SUM: 32.55 — highest summarization score across the entire leaderboard
• TQuAD breakthrough: +11.27 F1 improvement on Turkish reading comprehension
• Edge-deployable: Runs on a single consumer GPU, Mac Mini, or mobile device

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AlicanKiraz0/Kara-Kumru-v1.0-2B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Türkiye'nin en büyük gölü hangisidir ve özellikleri nelerdir?"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
)

inputs = {k: v.to(model.device) for k, v in inputs.items()}

output = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

response = tokenizer.decode(
    output[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True
)

print(response)

Quantized Inference (GGUF)

# For llama.cpp / Ollama users (if GGUF version is available)
ollama run AlicanKiraz0/Kara-Kumru-v1.0-2B

Training Details

Base Model

• Model: vngrs-ai/Kumru-2B
• Parameters: ~2B
• Architecture: Transformer decoder-only

Fine-tuning Configuration

• Method: Full fine-tuning
• Precision: BF16
• Hardware: SnakeEye Cluster (DGX Spark + Mac Studio M3 Ultra)

What Improved & Why

The fine-tuning primarily strengthened generative capabilities (QA, summarization, translation) while showing minor regression on some discriminative tasks (classification, NLI). This is a well-known trade-off in LLM fine-tuning — the model learned to produce better free-form Turkish text at the cost of some multiple-choice and classification accuracy.

Evaluation

All evaluations were performed using the Cetvel Turkish LLM Benchmark framework.

Cetvel Benchmark Categories

Intended Use

• Turkish question answering and information extraction
• Turkish text summarization
• Turkish-to-English translation
• General Turkish language generation
• Research on efficient Turkish LLMs

Limitations

• Classification tasks: Some regression on text classification and NLI compared to baseline
• Grammar correction: GEC performance decreased by ~3.4 points
• Model size trade-offs: While competitive with much larger models on generative tasks, MCQA performance lags behind 7B+ models
• Evaluation caveat: Cross-pipeline benchmark comparison — see note above

Roadmap (Kara-Kumru-v2.0)

• Targeted GEC and NLI distillation to recover regression
• Classification-focused fine-tuning (news categorization, irony detection)
• MCQA and causal reasoning dataset expansion
• Unified evaluation pipeline for fair cross-model comparison
• GGUF quantization for edge deployment

Citation

@misc{kiraz2026karakumru,
  title={Kara-Kumru-v1.0-2B: A Fine-tuned 2B Turkish LLM Outperforming 70B Models},
  author={Kiraz, Alican},
  year={2026},
  url={https://huggingface.co/AlicanKiraz0/Kara-Kumru-v1.0-2B}
}

Acknowledgments

• VNGRS AI for the Kumru base model and the Cetvel benchmark framework
• Built on the SnakeEye Cluster — a multi-node system with DGX Spark and Apple Silicon nodes

Contact

Alican Kiraz

---

Kara-Kumru (lit. "Dark Dove") — named after the darker variant of the Eurasian collared dove. Small but fierce.