Llama-3.1-8B-Turkish

Llama-3.1-8B-Turkish

Built with Llama. A Turkish language adaptation of meta-llama/Llama-3.1-8B, produced end-to-end on a single RTX 4090 (24 GB): tokenizer expansion → continued pretraining (QLoRA) → instruction fine-tuning. This is a research / portfolio model demonstrating the full adaptation pipeline, not a production system.

What was done

1. Tokenizer expansion — ~7–8k Turkish tokens injected into the Llama 3.1 (tiktoken-style) tokenizer; new embedding rows mean-initialized from existing sub-token embeddings.
2. Continued pretraining (CPT) — QLoRA (4-bit) on a cleaned Turkish corpus (Wikipedia + sampled/deduped mC4).
3. Instruction fine-tuning (SFT) — Turkish instruction data + targeted buckets (safety/refusals, constraint-following, math word-problems, and Turkish-specific skills: deasciification, morphology, idioms). Trains lm_head alongside LoRA to fix end-of-turn stopping.

Results vs meta-llama/Llama-3.1-8B-Instruct

Evaluated on held-out Turkish and recognized multilingual benchmarks (lm-evaluation-harness, 0-shot).

Headline wins (the axes a language-adaptation targets):

• Models Turkish ~14% better than Instruct (0.786 vs 0.912 bits/byte). Note: Instruct is worse than base on Turkish (0.912 vs 0.852) — a measurable alignment tax from English-centric RLHF.
• ~12% more token-efficient on Turkish (fewer tokens per word → cheaper inference, longer effective context).
• Competitive (statistical tie) with Instruct on Turkish commonsense (XCOPA) and NLI (XNLI).

Honest limitations: an 8B model cannot out-know a same-size, more heavily post-trained instruct model. This model trails Instruct on knowledge (MMLU) and reading comprehension (Belebele), and — like most small models — is unreliable at symbolic math and can hallucinate confidently. Use with a guardrail; not for high-stakes use.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

m = "erenzeytunn/Llama-3.1-8B-Turkish"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Türkiye'nin başkenti neresidir ve kısaca tarihinden bahset."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=True, temperature=0.3, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

License

Governed by the Llama 3.1 Community License. Built with Llama. Use is subject to Meta's Acceptable Use Policy.