Downloads · 30 days
15
10% of all-time downloads
abhi11nav/sakhi-telugu-681M-pretrained-0625
sakhi-telugu-681M-pretrained-0625 is a text generation model from abhi11nav. Use it when you need the model to write or continue text. The card lists the license as mit.
A transformer-based language model pretrained from scratch on a cleaned and deduplicated Telugu corpus. It is trained on high-quality, natural Telugu text collected from diverse sources.
Downloads · 30 days
15
10% of all-time downloads
All-time downloads
152
Public
Repo size
2.7 GB
Likes
0
Public
Click a slice to open those files.
.bin2.7 GB · 100%
From the Hugging Face model README
A transformer-based language model pretrained from scratch on a cleaned and deduplicated Telugu corpus. It is trained on high-quality, natural Telugu text collected from diverse sources.
MIT
te)text-generationThe training corpus was carefully prepared using the following steps to ensure data quality, linguistic relevance, and uniqueness:
A custom deduplication and cleaning pipeline was developed using MinHash and Locality Sensitive Hashing (LSH) to eliminate near-duplicate documents and maintain a diverse dataset.
Steps included:
Text Normalization:
MinHash-based Deduplication:
MinHashLSH index was used with:
num_perm = 128similarity_threshold = 0.95The Sakhi model was trained from scratch with the following configuration:
model_parameters:
embed_dim: 2048
num_heads: 8
ff_dim: 4096
chunk_length: 1024
num_layers: 10
vocab_size: 64000
The model was pretrained for 100 hours on 4× A100 GPUs provided by Lambda. Pretraining was done using PyTorch with mixed precision and DDP (DistributedDataParallel) for efficient scaling.
train_parameters:
batch_size: 12
num_epochs: 1
init_learning_rate: 1e-5
min_learning_rate: 1e-8
seed: 42
master_addr: "localhost"
master_port: "12355"
num_gpus: -1
save_every_n_steps: 25000
log_every_n_steps: 100
gradient_clipping_max_norm: 3.0
call_torch_compile_on_model: False
gradient_accumulation_steps: 2
🚀 GPU costs were partially sponsored by Lambda Labs.
paths:
tokenizer_path: "/"
dataset_path: "/"
save_dir: "/"
⚠️ Paths are placeholders — these should be replaced with actual paths