Downloads · 30 days
24
4% of all-time downloads
kk-llama-50m
Abzalbek89/kk-llama-50m
kk-llama-50m is a text generation model from Abzalbek89. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A small Kazakh language model (LLaMA architecture, ~50M parameters) trained from scratch on cleaned Kazakh text corpus.
All-time downloads
561
Public
Parameters
50.3M
201 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors201 MB · 98%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Apr 16, 2026
- Updated
- Apr 16, 2026
- SHA
- 6c1cebbe
Model card
From the Hugging Face model README
Kazakh LLaMA 50M
A small Kazakh language model (LLaMA architecture, ~50M parameters) trained from scratch on cleaned Kazakh text corpus.
Model Details
| Parameter | Value |
|---|---|
| Architecture | LLaMA |
| Parameters | ~50M |
| Hidden size | 576 |
| Layers | 8 |
| Attention heads | 8 |
| Vocab size | 32,000 |
| Context length | 2,048 tokens |
| Training tokens | ~0.51B (510M) |
| Precision | bf16 |
Training
| Hyperparameter | Value |
|---|---|
| Learning rate | 6e-4 |
| Scheduler | Cosine |
| Warmup steps | 500 |
| Batch size | 16 x 4 (grad accum) = 64 |
| Epochs | 3 |
| Weight decay | 0.1 |
| Max grad norm | 1.0 |
Results
| Metric | Value |
|---|---|
| Train loss | 3.38 |
| Eval loss | 3.379 |
| Perplexity | 29.35 |
Training Data
- Corpus: Abzalbek89/corpus_clean — 1.5M cleaned Kazakh texts
- Tokenized: Abzalbek89/corpus_clean_tokenized — 249K blocks x 2048 tokens
- Tokenizer: Abzalbek89/kk-tokenizer-bpe-32k — ByteLevel BPE, 32K vocab
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Abzalbek89/kk-llama-50m")
tokenizer = AutoTokenizer.from_pretrained("Abzalbek89/kk-llama-50m")
input_text = "Қазақстан Республикасының"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License
Apache 2.0
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- apache-2.0
- Languages
- kk
- Created
- Apr 16, 2026
- Updated
- Apr 16, 2026