Downloads · 30 days
23
11% of all-time downloads
srikarkashyap/telugu-gpt-moe-sangraha-10k
telugu-gpt-moe-sangraha-10k is a text generation model from srikarkashyap. Use it when you need the model to write or continue text. It is set up for pytorch. The card lists the license as cc-by-4.0.
This repository contains a 10,000-step base language-model checkpoint trained from scratch on Telugu text from the verified subset of ai4bharat/sangraha.
Downloads · 30 days
23
11% of all-time downloads
All-time downloads
201
Public
Parameters
298M
2.4 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors1.2 GB · 99%
From the Hugging Face model README
This repository contains a 10,000-step base language-model checkpoint trained from scratch on Telugu text from the verified subset of ai4bharat/sangraha.
It is a research/personal pretraining checkpoint, not an instruction-tuned assistant. Expect Telugu text continuation behavior, not reliable chat or question answering.
model.safetensorsThis is a custom PyTorch architecture. It loads through Hugging Face Auto classes with trust_remote_code=True.
ai4bharat/sangrahaverifiedtelThe Sangraha dataset card on Hugging Face lists the dataset license as cc-by-4.0.
Final train step:
step=10000
loss=2.9328
lm_loss=2.9224
router_aux_loss=1.0224
router_dropped_fraction=0.0151
router_entropy=1.0828
tokens_seen=327,680,000
Final validation result on the prepared local validation split:
eval_loss=2.9257
eval_lm_loss=2.9153
eval_perplexity=18.65
Eval perplexity trend:
500 84.64
1000 47.87
1500 36.22
2000 31.95
2500 29.52
3000 27.38
3500 25.95
4000 24.82
4500 23.57
5000 22.79
5500 22.06
6000 21.36
6500 20.81
7000 20.32
7500 19.87
8000 19.51
8500 19.24
9000 19.00
9500 18.75
10000 18.65
Evaluated on CohereForAI/Global-MMLU, config te, split test.
Four-pass full-option scoring with mean token log probability:
total=14,042
correct=3,400
accuracy=24.21%
This is essentially random-level for a 4-way benchmark. The model has learned some Telugu surface-form continuation, but it is not yet a useful knowledge or reasoning model.
Because this is a custom architecture, pass trust_remote_code=True:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "srikarkashyap/telugu-gpt-moe-sangraha-10k"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="cuda",
)
prompt = "తెలుగు భాష"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=96,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.15,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The model does not implement KV-cache yet, so generation works but is slower than stock GPT-style models with cached decoding.
Clone this repository and run the included sampler from the repo root:
uv run python sample_checkpoint.py `
--checkpoint . `
--device cuda `
--dtype bfloat16 `
--prompt "తెలుగు భాష" `
--max_new_tokens 96 `
--temperature 0.7 `
--top_p 0.9 `
--repetition_penalty 1.15
CPU sampling also works, but is slower:
uv run python sample_checkpoint.py --checkpoint . --device cpu --prompt "తెలుగు భాష"
This checkpoint is best used as:
It should not be used for production factual answering, medical/legal/financial advice, or safety-critical applications.
Training data came from ai4bharat/sangraha. Evaluation used CohereForAI/Global-MMLU.